Analyzing server problems using jvisualvm

Problem description ： A game area , No one can log in , Front end display loading 了 , The back end did not report an error .

Memory evidence collection ： First of all, you should catch the memory information of the server , adopt windows The task manager finds the pid, Then open the jvisualvm, Find the corresponding pid application , Click on “ Threads ” label , Click on “ Threads dump”, You can see the working status of all internal threads of the reference program .

Then start to analyze ,

As shown in the figure above , There are multiple login threads stuck run.MemcachedRun.add(MemcachedRun.java:97) here , These login threads are waiting for the lock <0x000000078004a720>, therefore , Let's see which thread holds the lock .

As shown in the figure above ,<0x000000078004a720> The owner of this lock is also waiting for the release of another lock , This lock is <0x000000078004a6d8>, However , In the print thread dump The lock cannot be found in the message , Prove that this lock is java Internal locks , Not our own code .

So it's positioned to run.MemcachedRun.delSynObject(MemcachedRun.java:341) Look at this line of code , The code is as follows ：

SynObject syn=Link.take();

This Link The definition of public static BlockingQueue<SynObject> Link = new LinkedBlockingQueue<SynObject>();

So , This lock is the lock of the system , To verify this system lock , Specially wrote a section of test code , as follows ：

package utils.mytest;

import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

import MyObj.SynObject;

public class cerun  extends Thread{
	public static BlockingQueue<SynObject> Link = new LinkedBlockingQueue<SynObject>();
	public static void main(String[] strs){
		cerun cerun=new cerun();
		cerun.start();
		for(int i=0;i<10;i++){
			try {
				System.out.println(" Start the first "+i+" Time , Data volume ="+cerun.Link.size());
				if(cerun.Link.size()>0){
					// Restart thread 
					cerun.stop();
					Thread.sleep(1000);
					System.out.println(" Restart thread ");
					cerun=new cerun();
					cerun.start();
					// Delete data 
					cerun.delSynObject(1);
				}else{
					System.out.println(" No data ");
				}
				Thread.sleep(2);
			} catch (Exception e) {
				e.printStackTrace();
			}
		}
	}
	public cerun(){
		
	}
	public void run() {
		while(true){
			try {
				for(int i=0;i<10000;i++){
					SynObject SynObject=new SynObject();
					SynObject.setId(1);
					Link.offer(SynObject);
				}
				if(Link.size()>2000000){
					System.out.println(" The thread ends itself ");
					break;
				}
			} catch (Exception e1) {
				e1.printStackTrace();
			}
		}
	}
	public void delSynObject(int id){
		for(SynObject object : Link){
			if(object.getId()==id){
				Link.remove(object);
			}
		}
	}
}

Run the above code , Output is as follows ：

 Start the first 0 Time , Data volume =0
 No data 
 Start the first 1 Time , Data volume =2991
 Restart thread 

Then no output , Prove deadlock , see jvm Content , The important contents are as follows ：
 

"main" prio=6 tid=0x0000000003303800 nid=0x3258 waiting on condition [0x00000000032ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007d5af4d30> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at java.util.concurrent.LinkedBlockingQueue.fullyLock(LinkedBlockingQueue.java:223)
        at java.util.concurrent.LinkedBlockingQueue$Itr.<init>(LinkedBlockingQueue.java:792)
        at java.util.concurrent.LinkedBlockingQueue.iterator(LinkedBlockingQueue.java:778)
        at utils.mytest.cerun.delSynObject(cerun.java:55)
        at utils.mytest.cerun.main(cerun.java:24)

   Locked ownable synchronizers:
        - None
		
"Thread-1" prio=6 tid=0x000000000d9a6800 nid=0x1ed4 waiting on condition [0x000000000df3f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007d5af4d30> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:417)
        at utils.mytest.cerun.run(cerun.java:43)

   Locked ownable synchronizers:
        - None

You can find main Threads and created cerun Threads are waiting for locks 0x00000007d5af4d30, and 0x00000007d5af4d30 Where have you been? , Threads dump Can't find it inside .