Clash Royale CLAN TAG

Python process in loop or re-scheduled memory leak

I have a function running inside a dedicated multiprocessing process, which can run endlessly in a loop, re-scheduled itself using threading timer or run only one time. In the first and second case, the memory usage is endlessly leaking, even though I am just updating the references to my variables in the endless loop mode. I use a queue for communication with the main process and pop everything out of it right away from the main process.

Here is the function code :

def schedule(args, fn, q = None ):

isendless = True
while isendless:

# retrieve from database
pd_original_matrice = get_rating(args);

# save userID / index correspondance
index = pd_original_matrice['CustomerId'].values
index = dict( zip(index, list(range(0, len(index))) ))

now = dt.now().replace(microsecond=0) # start timer

mf = MF(pd_original_matrice.iloc[:,1:].fillna(0).values.astype(np.float64)
, args.nbFeatures, args.learning_rate, args.regularization, args.nbEpoch)

training_rmse, testing_rmse = mf.train(args.prediction_method, args.factorization_method, split=args.split, reduc=args.reduc)

then = dt.now().replace(microsecond=0) # get elapsed time
log.normal('__MF has ended, duration %s , min test RMSE of %0.6f achieved at epoch %d'
%( str(then-now), min(testing_rmse), testing_rmse.index(min(testing_rmse))+1 ))

# schedule a new MF
if args.cycle != 0:
log.normal('___New MF schedule in %d seconds' %args.cycle)
t = timer(args.cycle, schedule, [args, fn, q]) # re-schedule
t.start()
print(Fore.WHITE)

if (q != None): # for endless or re-scheduled
q.put( (mf, pd_original_matrice, index) )

else: # For one-time mf
return mf, pd_original_matrice, index

if args.endless_mf != 1: # final do while test
isendless = False


The memory taken by the functions called inside this process should be free as well, and I believe that in the re-schedule mode the process would terminate after scheduling a new thread with threading.timer . Do you have any ideas ? I thought the garbage collector would delete object without any more references and free up memory. Is it related to a misunderstanding of the garbage collector's behavior ?

Thanks

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.