Celery takes up large memory - memory leak

Problem finding

Originally top Command to view the real-time memory of the server ,  Suddenly found that 32G Memory celery Occupy to 12%,  It takes up a lot of memory 4G 了 .

1. ​​top​​​ Under orders ,​​M​​​ Key on press ​​MEM​​​ Column sorted process list , Press ​​H​​​ Then view the thread list , because ​​worker​​ No child threads , So only one record is displayed ;
2. Press ​​c​​​ command , Start ​​COMMAND​​ List details , You can find the corresponding process initiator ;
3. Press ​​o​​​ key , Turn on the filtering function , Input ​​COMMAND=celery​​​ Display only ​​celery​​ List of processes named ;

reason

By default ,celery Of worker Is the maximum number of tasks that the work process can execute before being replaced by a new task ,  The default is unlimited .

see ​ ​ Website shows ​​

Log in and copy

      
      

       
       """
       
       

       
       celery To configure 
       
       

       
       """
       
       

       
       

       
       from 
       
       celery.
       
       schedules 
       
       import 
       
       crontab
       
       

       
       

       
       #  A middleman 
       
       

       
       BROKER_URL 
       
       = 
       
       'redis://:[email protected]/x'
       
       

       
       

       
       worker_prefetch_multiplier 
       
       = 
       
       4  
       
       # celery worker Concurrency number 
       
       

       
       worker_max_tasks_per_child 
       
       = 
       
       2  
       
       #  Every worker Maximum number of tasks performed 
       
       

       
       

       
       

       
       #  Designated time zone , Do not specify default as  'UTC'
       
       

       
       CELERY_TIMEZONE 
       
       = 
       
       'Asia/Shanghai'
       
       

       
       

       
       #  Specify the imported task module 
       
       

       
       CELERY_IMPORTS 
       
       = (
       
       
    
       
       'celery_app.tasks',
       
       
)
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
       
       
12.
       
       
13.
       
       
14.
       
       
15.
       
       
16.
       
       
17.
       
       
18.
       
       
19.
       
       
20.
      
      

Need configuration celery Set up But the parameters written on the official website have errors in the execution process " The new setting name cannot be mixed with the old setting name "

Cannot mix new setting names with old setting names, please
rename the following settings to use the old format:

worker_max_tasks_per_child           -> CELERYD_MAX_TASKS_PER_CHILD
worker_prefetch_multiplier           -> CELERYD_PREFETCH_MULTIPLIER

The configuration name needs to be modified :

Log in and copy

      
      

       
       """
       
       

       
       celery To configure 
       
       

       
       """
       
       

       
       

       
       from 
       
       celery.
       
       schedules 
       
       import 
       
       crontab
       
       

       
       

       
       #  A middleman 
       
       

       
       BROKER_URL 
       
       = 
       
       'redis://:[email protected]/x'
       
       

       
       

       
       CELERYD_PREFETCH_MULTIPLIER 
       
       = 
       
       4  
       
       # celery worker Concurrency number 
       
       

       
       CELERYD_MAX_TASKS_PER_CHILD 
       
       = 
       
       2  
       
       #  Every worker Maximum number of tasks performed 
       
       

       
       

       
       #  Designated time zone , Do not specify default as  'UTC'
       
       

       
       CELERY_TIMEZONE 
       
       = 
       
       'Asia/Shanghai'
       
       

       
       

       
       #  Specify the imported task module 
       
       

       
       CELERY_IMPORTS 
       
       = (
       
       
    
       
       'celery_app.tasks',
       
       
)
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
       
       
7.
       
       
8.
       
       
9.
       
       
10.
       
       
11.
       
       
12.
       
       
13.
       
       
14.
       
       
15.
       
       
16.
       
       
17.
       
       
18.
       
       
19.
      
      

Restart

perform

Log in and copy

      
      

       
       celery beat -A celery_app -l info --detach
      
      
      
      

       
       
1.
      
      

start-up celery, adopt ps -ef | grep celery You can see two celery worker process （8226,8228）

utilize celery worker Carry out a task , When worker When the maximum task is not performed （ Destroy and rebuild ）, Each time a task is executed, the memory occupied will inevitably increase , The number of tasks is 9,10 when （celery Uniform scheduling , Concurrency number * Maximum number of tasks ）, There are original 8228 worker Be destroyed , Recreate 9386 worker And the original 8226 worker Be destroyed , Recreate 9564 worker, here , Run the 9 When the time , The total memory consumption has decreased , Run the 10 When the time , The total memory returns to the initial value , Carry out the same task 19、20 The situation is similar .

celery Concurrent calculation rules
celery Task concurrency is only associated with celery Configuration item CELERYD_CONCURRENCY of , And CELERYD_MAX_TASKS_PER_CHILD It doesn't matter. , namely CELERYD_CONCURRENCY=2, Only concurrent 2 individual worker, At this time, when the task processes large files , Execute twice to see two task Tasks are executed in parallel , While performing the third task , Start queuing , Until two worker completion of enforcement .

Conclusion
celery After executing the task, do not release the memory and the original worker It has not been destroyed , therefore CELERYD_MAX_TASKS_PER_CHILD Small points can be configured appropriately , The number of concurrent tasks is the same as CELERYD_CONCURRENCY Configuration item , Every time you add one worker Inevitably increase memory consumption , It also affects a worker When will it be destroyed , because celery It is to evenly schedule tasks to each worker, Therefore, the configuration should not be too large , Properly configured .