当前位置:网站首页>On December 8th, 2020, the memory of marketing MRC application suddenly increased, resulting in system oom
On December 8th, 2020, the memory of marketing MRC application suddenly increased, resulting in system oom
2022-07-07 08:57:00 【bboyzqh】
List of articles
background
12.08 At noon on the th mrc Applications suddenly appear, memory continues to rise , From 67% Rise to 85% about ( Monitoring is as follows ), Fortunately, the rising process is relatively slow , A decisive restart solved the problem . The process of solving and analyzing problems is as follows .
Problem solving process
mrc It's the bottom application of marketing , Main partial rule calculation , common 6 Taiwan machine (2 Next cluster , And cluster traffic is isolated from each other , Such as the upper layer hipc Cluster traffic will not be requested to k8s Cluster machines ),6 At the same time, the memory keeps rising , Refer to sketch 1 .
Because it was a big promotion at noon that day , Considering that there are only 3 Taiwan machine , I'm afraid that in the process of restarting one , The other two can't stand the flow of big promotion , At first, I didn't dare to consider a single restart , After a short period of time, the decision was made taking into account the cpu Only 5% about , The worst worry is that memory can't take care of it all of a sudden , If frequent gc May affect the normal traffic access , So prepare for the worst : Restart decisively ( Remove traffic before restart , meanwhile dump Memory for subsequent analysis ), As a result, there was no problem , Refer to sketch 2 . The whole process is as follows :
- The target restarts the machine for traffic removal , Adjust to restart the machine dubbo The weight of 0 that will do , because dump Memory processes are memory consuming operations , Server may appear feign death phenomenon, affect normal call , So we need to remove the traffic .
- Force the target machine to restart once full gc, The purpose is to reclaim the normal memory object occupation , To prevent the normal memory occupation and the influence of real memory leak objects , The impact analysis , You can use the following command :
- dump Next target machine memory , The order is as follows :
jmap -histo:live 13 ( Trigger full gc)
or
jmap -dump:live,file=dump_001.bin 13 ( Trigger full gc, When triggered, put dump_001.bin File deletion )
or
jcmd 13 GC.run ( Trigger young gc)
- Use IBMAnalyzer( perhaps jdk Self contained jvisualvm Tools or mat Tools ) Yes dump File analysis is enough
jmap -dump:format=b,file=dumpFile 13
After the event, the best plan is to add a new one to Tongyun maintenance mrc machine , And then restart each one , Refer to sketch 3 .
Post analysis
After the event dump Document analysis , As it involves specific business, I will not elaborate on it , Just describe the conclusion : Because that day mrc Configuring the shadow library results in . The root cause is druid Threads that monitor shadow library configuration will not exit with the end of the pressure test , stay mrc After pressure testing, the thread creation is triggered without restart , Lead to mrc Application memory keeps rising .
Welcome to WeChat official account. : Fang Chen's blog
边栏推荐
- Greenplum 6.x monitoring software setup
- Quick sorting (detailed illustration of single way, double way, three way)
- Alibaba P8 teaches you how to realize multithreading in automated testing? Hurry up and stop
- RuntimeError: Calculated padded input size per channel: (1 x 1). Kernel size: (5 x 5). Kernel size c
- Ppt template and material download website (pure dry goods, recommended Collection)
- Un salaire annuel de 50 W Ali P8 vous montrera comment passer du test
- 面试题:高速PCB一般布局、布线原则
- Why choose cloud native database
- Troublesome problem of image resizing when using typora to edit markdown to upload CSDN
- 平台化,强链补链的一个支点
猜你喜欢
Led analog and digital dimming
数字三角形模型 AcWing 275. 传纸条
Other 7 features of TCP [sliding window mechanism ▲]
Greenplum 6.x common statements
Three series of BOM elements
Panel display technology: LCD and OLED
ncs成都新電面試經驗
LED模拟与数字调光
Troublesome problem of image resizing when using typora to edit markdown to upload CSDN
Ppt template and material download website (pure dry goods, recommended Collection)
随机推荐
How to count the number of project code lines
为不同类型设备构建应用的三大更新 | 2022 I/O 重点回顾
[Yugong series] February 2022 U3D full stack class 006 unity toolbar
指针进阶,字符串函数
Simulation volume leetcode [general] 1567 Length of the longest subarray whose product is a positive number
Simulation volume leetcode [general] 1705 The maximum number of apples to eat
Calf problem
Image segmentation in opencv
xray的简单使用
Greenplum6.x常用语句
2022-06-30 unity core 8 - model import
MySQL主从延迟的解决方案
FPGA knowledge accumulation [6]
GoLand set goproxy
[Yugong series] February 2022 U3D full stack class 008 - build a galaxy scene
H3C VXLAN配置
Several methods of calculating the average value of two numbers
Test pits - what test points should be paid attention to when adding fields to existing interfaces (or database tables)?
LeetCode 736. LISP syntax parsing
Greenplum6.x搭建_安装