JVM problem replication

One 、 Problem description
Business scenario ： Distributed timed tasks , Gray scale test reconstructed code , The original code processing time is 90-100 minute , drawn-out , Refactoring new code , The target processing time is 40 minute

The phenomenon ： During a mission ,cpu continued 80-90%, The system load is as high as 30 above （32 nucleus cpu）, The usual load is 7 about , When the task is executed , There's a megacard , All other programs deployed on the current machine are affected , forced , restart app , After reboot , Back to normal , But the next day the situation can still be repeated

Two 、 The screening process
Step one
Relevant business code has been reconstructed , When something goes wrong , Using new code , It is preliminarily determined that there is a problem with the suspected code, causing related problems
Solution ： Drop the new code , again code review
Conclusion ： It's not a code problem

Step two ： guess jvm There may be problems ,full gc
When the problem reappears , Print related gc journal

Found every three seconds full gc, Lead to cpu Keep rising
The judgment question is full gc The frequency is too high

3、 ... and 、 The analysis reason
Business code refactoring leads to full gc Very high frequency ,code review And then I found out , The new code will produce a large number of objects , These objects should be in ygc To be recycled , But the opposite is true ,ygc No recycling , Memory objects continue to be created in older generations , Cause high frequency but full gc produce , The suspected memory guarantee mechanism leads to this problem
Check jvm Parameters
This project has been running for three years , But no collector is configured - - ！

Four 、 modify jvm Parameters
At present jvm Memory 6gb, So we use it directly g1 The collector
Configuration related jvm Parameters , And observe jvm Memory and cpu usage
jvm The parameters are as follows

-Xmx6g -Xms6g  -Xss256k  -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=128m -Xloggc:/usr/local/logs/server/tomcat_gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/logs/heapdump/server.dump  -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps  -Dcom.sun.management.jmxremote.port=19001 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname= Server public network ip

In the use of g1 after , Compare the original code with the new code ,jvm The operation is as follows
Original code ：

New code ：


5、 ... and 、 Conclusion ：
Use g1 The collector acts as a garbage collector for the project

remarks ：
New code will continue to trigger gc,g1 Of ygc Time is very short , In about 30-40ms Between , Business can tolerate ,cpu There will still be some jitters , But it doesn't affect other applications ,cpu The utilization rate is controllable , Business processing time is 37 minute , Code refactoring can meet the corresponding business requirements

The resumption is over