当前位置：网站首页>Celery takes up large memory - memory leak

Celery takes up large memory - memory leak

2022-07-26 06:49:00 【51CTO】

Problem finding

Originally top Command to view the real-time memory of the server , Suddenly found that 32G Memory celery Occupy to 12%, It takes up a lot of memory 4G 了 .

top Under orders ,M Key on press MEM Column sorted process list , Press H Then view the thread list , because worker No child threads , So only one record is displayed ;
Press c command , Start COMMAND List details , You can find the corresponding process initiator ;
Press o key , Turn on the filtering function , Input COMMAND=celery Display only celery List of processes named ;

reason

By default ,celery Of worker Is the maximum number of tasks that the work process can execute before being replaced by a new task , The default is unlimited .
see Website shows

       
       """
       
       celery To configure 
       
       """
       
       from 
       
       celery.
       
       schedules 
       
       import 
       
       crontab
       
       #  A middleman 
       
       BROKER_URL 
       
       = 
       
       'redis://:[email protected]/x'
       
       worker_prefetch_multiplier 
       
       = 
       
       4  
       
       # celery worker Concurrency number 
       
       worker_max_tasks_per_child 
       
       = 
       
       2  
       
       #  Every worker Maximum number of tasks performed 
       
       #  Designated time zone , Do not specify default as  'UTC'
       
       CELERY_TIMEZONE 
       
       = 
       
       'Asia/Shanghai'
       
       #  Specify the imported task module 
       
       CELERY_IMPORTS 
       
       = (
       
       'celery_app.tasks',
       
)
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.

Need configuration celery Set up But the parameters written on the official website have errors in the execution process " The new setting name cannot be mixed with the old setting name "

Cannot mix new setting names with old setting names, please
rename the following settings to use the old format:

worker_max_tasks_per_child -> CELERYD_MAX_TASKS_PER_CHILD
worker_prefetch_multiplier -> CELERYD_PREFETCH_MULTIPLIER

The configuration name needs to be modified :

       
       """
       
       celery To configure 
       
       """
       
       from 
       
       celery.
       
       schedules 
       
       import 
       
       crontab
       
       #  A middleman 
       
       BROKER_URL 
       
       = 
       
       'redis://:[email protected]/x'
       
       CELERYD_PREFETCH_MULTIPLIER 
       
       = 
       
       4  
       
       # celery worker Concurrency number 
       
       CELERYD_MAX_TASKS_PER_CHILD 
       
       = 
       
       2  
       
       #  Every worker Maximum number of tasks performed 
       
       #  Designated time zone , Do not specify default as  'UTC'
       
       CELERY_TIMEZONE 
       
       = 
       
       'Asia/Shanghai'
       
       #  Specify the imported task module 
       
       CELERY_IMPORTS 
       
       = (
       
       'celery_app.tasks',
       
)
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.

Restart

perform

       
       celery beat -A celery_app -l info --detach
      
1.

start-up celery, adopt ps -ef | grep celery You can see two celery worker process （8226,8228）

utilize celery worker Carry out a task , When worker When the maximum task is not performed （ Destroy and rebuild ）, Each time a task is executed, the memory occupied will inevitably increase , The number of tasks is 9,10 when （celery Uniform scheduling , Concurrency number * Maximum number of tasks ）, There are original 8228 worker Be destroyed , Recreate 9386 worker And the original 8226 worker Be destroyed , Recreate 9564 worker, here , Run the 9 When the time , The total memory consumption has decreased , Run the 10 When the time , The total memory returns to the initial value , Carry out the same task 19、20 The situation is similar .

celery Concurrent calculation rules
celery Task concurrency is only associated with celery Configuration item CELERYD_CONCURRENCY of , And CELERYD_MAX_TASKS_PER_CHILD It doesn't matter. , namely CELERYD_CONCURRENCY=2, Only concurrent 2 individual worker, At this time, when the task processes large files , Execute twice to see two task Tasks are executed in parallel , While performing the third task , Start queuing , Until two worker completion of enforcement .

Conclusion
celery After executing the task, do not release the memory and the original worker It has not been destroyed , therefore CELERYD_MAX_TASKS_PER_CHILD Small points can be configured appropriately , The number of concurrent tasks is the same as CELERYD_CONCURRENCY Configuration item , Every time you add one worker Inevitably increase memory consumption , It also affects a worker When will it be destroyed , because celery It is to evenly schedule tasks to each worker, Therefore, the configuration should not be too large , Properly configured .

原网站

版权声明
本文为[51CTO]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/201/202207181833374598.html

当前位置：网站首页>Celery takes up large memory - memory leak

Celery takes up large memory - memory leak

边栏推荐

猜你喜欢

随机推荐