[JVM optimization ultra detailed] common JVM tuning scenarios

cpu Occupy too much

cpu If the occupation is too high, it should be discussed according to the situation , Is there a business activity , Suddenly a lot of traffic came in , And after the event cpu The occupancy rate fell , If this is the case, in fact, you can not care too much , Because the more requests , The more threads to process , This is a normal phenomenon . Come back , If your server configuration itself is poor ,cpu There is only one core , This situation , A little more traffic can really put your cpu Run out of resources , At this time, we should consider upgrading the configuration first .

The second case ,cpu The occupancy rate is too high for a long time , In this case, it may be that your program has code with super many cycles , Even a dead circle . The troubleshooting steps are as follows ：

（1） use top Command view cpu Occupancy

So you can locate cpu Too much progress . stay linux Next ,top Command to get the process number and jps Tools get vmid It's the same ：

（2） use top -Hp View the command of thread

You can see that it's a thread id by 7287 This thread has been occupying cpu

（3） Convert the thread number to 16 Base number

[[email protected] ~]# printf "%x" 7287
1c77

Write this down 16 Decimal digit , Now we're going to use

（4） use jstack Tool to check the thread stack

[[email protected] ~]# jstack 7268 | grep 1c77 -A 10
"http-nio-8080-exec-2" #16 daemon prio=5 os_prio=0 tid=0x00007fb66ce81000 nid=0x1c77 runnable [0x00007fb639ab9000]
   java.lang.Thread.State: RUNNABLE
 at com.spareyaya.jvm.service.EndlessLoopService.service(EndlessLoopService.java:19)
 at com.spareyaya.jvm.controller.JVMController.endlessLoop(JVMController.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)

adopt jstack The tool outputs the current thread stack , Re pass grep The command is combined with the thread obtained in the previous step 16 It's binary id Locate the operation of this thread , among jstack hinder 7268 It's No （1） The process number located in step ,grep The back is （2）、（3） The thread number located in step .

From the output, you can see that the thread is running , In execution com.spareyaya.jvm.service.EndlessLoopService.service This method , The code line number is 19 That's ok , So you can go to the code 19 That's ok , Find the code block where it is located , See if it's in a loop , In this way, the problem is located .

Deadlock

Deadlock is not as obvious as the first scenario ,web The application must be a multithreaded program , It serves multiple requests , After the program deadlocks , The deadlock thread is waiting （WAITING or TIMED_WAITING）, Threads in waiting state do not occupy cpu, The memory consumption is also very limited , And the performance may be that the request cannot be made , Finally, it timed out . When there are few deadlocks , This situation is not easy to find .

have access to jstack Tools to view

（1）jps see java process

[[email protected] ~]# jps -l
8737 sun.tools.jps.Jps
8682 jvm-0.0.1-SNAPSHOT.jar

（2）jstack View deadlock issues

because web Applications tend to have many worker threads , Especially in the case of high concurrency, the number of threads is more , So the output of this command will be very much .jstack The biggest advantage is that the deadlock information （ What thread generated the inclusion ） Output to last , So we just need to see the final content

Java stack information for the threads listed above:
===================================================
"Thread-4":
 at com.spareyaya.jvm.service.DeadLockService.service2(DeadLockService.java:35)
 - waiting to lock <0x00000000f5035ae0> (a java.lang.Object)
 - locked <0x00000000f5035af0> (a java.lang.Object)
 at com.spareyaya.jvm.controller.JVMController.lambda$deadLock$1(JVMController.java:41)
 at com.spareyaya.jvm.controller.JVMController$$Lambda$457/1776922136.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:748)
"Thread-3":
 at com.spareyaya.jvm.service.DeadLockService.service1(DeadLockService.java:27)
 - waiting to lock <0x00000000f5035af0> (a java.lang.Object)
 - locked <0x00000000f5035ae0> (a java.lang.Object)
 at com.spareyaya.jvm.controller.JVMController.lambda$deadLock$0(JVMController.java:37)
 at com.spareyaya.jvm.controller.JVMController$$Lambda$456/474286897.run(Unknown Source)
 at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Found a deadlock , The reason is also clear at a glance .

Memory leak

We all know ,java and c++ The biggest difference is that the former will automatically reclaim the memory that is no longer used , The latter requires the programmer to manually release . stay c++ in , If we forget to free memory, a memory leak will occur . however , Don't assume that jvm Help us recycle the memory and there will be no memory leak .

After the program has a memory leak , The available memory of the process will slowly decrease , The end result is to throw OOM error . happen OOM After the error, you may think that the memory is not large enough , So the -Xmx Increase the parameter , Then restart the app . The result is , After a while ,OOM There will still be . Finally, you can't increase the maximum heap memory , The result is that you can only restart the application at regular intervals .

Another possible manifestation of a memory leak is that the response time of the request becomes longer . This is because of frequent GC Will pause all other threads （Stop The World） Caused by the .

To simulate the scene , The following procedure is used

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class Main {
    public static void main(String[] args) {
        Main main = new Main();
        while (true) {
            try {
                Thread.sleep(1);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            main.run();
        }
    }

    private void run() {
        ExecutorService executorService = Executors.newCachedThreadPool();
        for (int i = 0; i < 10; i++) {
            executorService.execute(() -> {
                // do something...
            });
        }
    }
}

The operating parameters are -Xms20m -Xmx20m -XX:+PrintGC, Turn down the available memory , And it's happening gc Output information when , The operation results are as follows

...
[GC (Allocation Failure)  12776K->10840K(18432K), 0.0309510 secs]
[GC (Allocation Failure)  13400K->11520K(18432K), 0.0333385 secs]
[GC (Allocation Failure)  14080K->12168K(18432K), 0.0332409 secs]
[GC (Allocation Failure)  14728K->12832K(18432K), 0.0370435 secs]
[Full GC (Ergonomics)  12832K->12363K(18432K), 0.1942141 secs]
[Full GC (Ergonomics)  14923K->12951K(18432K), 0.1607221 secs]
[Full GC (Ergonomics)  15511K->13542K(18432K), 0.1956311 secs]
...
[Full GC (Ergonomics)  16382K->16381K(18432K), 0.1734902 secs]
[Full GC (Ergonomics)  16383K->16383K(18432K), 0.1922607 secs]
[Full GC (Ergonomics)  16383K->16383K(18432K), 0.1824278 secs]
[Full GC (Allocation Failure)  16383K->16383K(18432K), 0.1710382 secs]
[Full GC (Ergonomics)  16383K->16382K(18432K), 0.1829138 secs]
[Full GC (Ergonomics) Exception in thread "main"  16383K->16382K(18432K), 0.1406222 secs]
[Full GC (Allocation Failure)  16382K->16382K(18432K), 0.1392928 secs]
[Full GC (Ergonomics)  16383K->16382K(18432K), 0.1546243 secs]
[Full GC (Ergonomics)  16383K->16382K(18432K), 0.1755271 secs]
[Full GC (Ergonomics)  16383K->16382K(18432K), 0.1699080 secs]
[Full GC (Allocation Failure)  16382K->16382K(18432K), 0.1697982 secs]
[Full GC (Ergonomics)  16383K->16382K(18432K), 0.1851136 secs]
[Full GC (Allocation Failure)  16382K->16382K(18432K), 0.1655088 secs]
java.lang.OutOfMemoryError: Java heap space

You can see that although it has been gc, It takes up more and more memory , Some objects of the program cannot be recycled . But the above program objects are defined in methods , It's a local variable , Local variables after the result of the method run , The referenced object is in gc Should be recycled , But there is obviously no .

In order to find out which objects can't be recycled , We add the operating parameters -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=heap.bin, It means that OOM Put the heap memory information dump come out . Run the program until it is abnormal , So I got heap.dump file , And then we use eclipse Of MAT Plug in to analyze , If not, you need to install first .

then File->Open Heap Dump... , Then choose just now dump The document came out , choice Leak Suspects

MAT All objects that may have memory leaks are listed

You can see that there are 21260 individual Thread object ,3386 individual ThreadPoolExecutor object , If you go and have a look java.util.concurrent.ThreadPoolExecutor Source code , It can be found that the thread pool is used to reuse threads , Will constantly wait for new tasks , Threads do not recycle , It needs to be called shutdown() Method to stop the thread pool after executing the task .

In fact, thread pools are defined as local variables , A good practice is to set it as a singleton .

The above is just one of the methods

Online applications , Memory is often set very large , This happens OOM Then take the memory snapshot dump The file will be very big , It may be too big to analyze in the local computer （ Because there is not enough memory to open this dump file ）. Here is another solution ：

（1） use jps Navigate to process number

C:\Users\spareyaya\IdeaProjects\maven-project\target\classes\org\example\net>jps -l
24836 org.example.net.Main
62520 org.jetbrains.jps.cmdline.Launcher
129980 sun.tools.jps.Jps
136028 org.jetbrains.jps.cmdline.Launcher

Because we already know which application happened OOM, This can be used directly jps Find the process number 135988

（2） use jstat analysis gc Activity situation

jstat It's a statistic java Process memory usage and gc Active tools , There can be many parameters , Can pass jstat -help View all parameters and their meanings

C:\Users\spareyaya\IdeaProjects\maven-project\target\classes\org\example\net>jstat -gcutil -t -h8 24836 1000
Timestamp         S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
           29.1  32.81   0.00  23.48  85.92  92.84  84.13     14    0.339     0    0.000    0.339
           30.1  32.81   0.00  78.12  85.92  92.84  84.13     14    0.339     0    0.000    0.339
           31.1   0.00   0.00  22.70  91.74  92.72  83.71     15    0.389     1    0.233    0.622

The above is a command, which means output gc The situation of , Output time , Every time 8 Line output a line header , The statistical process number is 24836, Every time 1000 Output information once in milliseconds .

The output message is Timestamp It's distance jvm Start time ,S0、S1、E Are two of the new generation Survivor and Eden,O It's the senior generation area ,M yes Metaspace,CCS Use compression ratio ,YGC and YGCT They are the new generation gc Times and times of ,FGC and FGCT They are the old age gc Times and times of ,GCT yes gc The total time of . Although it happened gc, But the memory occupancy rate of the elderly generation has not decreased at all , It means that some objects cannot be recycled （ Of course, it doesn't rule out that these objects are really useful ）.

（3） use jmap Tools dump Out of memory snapshot

jmap You can assign java Memory snapshot of process dump come out , The effect is the same as the first treatment , The difference is that it doesn't have to wait OOM You can do that , and dump The snapshot will be much smaller .

jmap -dump:live,format=b,file=heap.bin 24836

And then you get heap.bin Memory snapshot file of , And then you can use it eclipse To analyze .

summary

Strictly speaking, the above three are not jvm Tuning of , Just use the jvm The tool finds out the problems in the code . We carry out jvm The main purpose of is to minimize pause time , Improve system throughput .

But if we don't analyze the system, we set the parameters blindly , May get worse results ,jvm To this day , Various default parameters may be balanced by people in the laboratory after many tests , Suitable for most application scenarios .

If you think your jvm Tuning is really necessary , It is also necessary to take samples for analysis , Finally, it has to be adjusted slowly and many times , It is possible to get better results .