Jstack stuff

Preface

Worked for many years , Often encounter supernatural events , for instance : " Why doesn't the task run " , " There is no complicated business , Yes? cpu So high " ," Used cache , Why is the query still so slow " , Some students who have worked for a year or two may be at a loss when encountering such problems , So I want to write some words to share with you . Writing co., LTD. , Mainly communication . This article is to communicate with you first jstack Usage of , And some suggestions on programming .

1 Classic scenes

• Scene one ： Mission （ Threads ） Suddenly stopped running The program is a simple quartz Mission , Pseudo code is similar to :

public class DataJobs {
   public void run() {
    log.info(" The task begins to run ");
    // Prepare the data  
    Object prepareData = prepare();
    // call http Remote interface acquisition 
    Result result = getHttpRemoteData(prepareData);
    // Storage 
    store(result);
    log.info(" The task ends running ");
   }
}

Copy

You can view the log to “ The task begins to run ” But no subsequent printing “ The task ends running ”

Business is not handled normally , There are no abnormalities In desperation I have to restart the service

• Scene two cpu load Very high The business is not complicated

Here's the picture :

The business scenario is live score .

The logic of the page is every few seconds from memached Get real-time game information , Get data according to color category , Then the front-end rendering .

Page access Pseudo code is similar to :

 @RequestMapping("/getLiveData")
 @ResponseBody
 public String getLiveData(String lotteryCode) {
     // from memcache Get score live data in    Just one  get key from memcahed
     String result = getDataFromMemcache(lotteryCode); 
     return result;
 }

Copy

Abnormal problems ： The page is very catchy , Server cpu utilization and load Very high .

2 jstack analysis

jstack yes jdk Built in thread stack analysis tool , Use this command to view or export Java Thread stack information in the application .

Executing commands is similar :

jstack -l pid >> statck.txt

Copy

Sometimes You can also use kill -3 command Achieve the effect of printing stack

kill  -3  pid

Copy

Stack format ：

"redisson-netty-2-4" #37 prio=5 os_prio=0 tid=0x00007f91ae738000 nid=0x3bf1 runnable [0x00007f91a5b4d000]
  java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked <0x00000000ee106580> (a io.netty.channel.nio.SelectedSelectionKeySet)
      - locked <0x00000000ee1065b0> (a java.util.Collections$UnmodifiableSet)
      - locked <0x00000000edff8448> (a sun.nio.ch.EPollSelectorImpl)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
      at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:752)
       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
       at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
       at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

Copy

dump Format specification ：

attribute	explain
redisson-netty-2-4	Thread name
prio=5	This thread jvm The priority ‍
os_prio=0	The thread is in os Priority in ‍
tid	The jvm Internal thread id ‍
nid	Native thread id Local operating system related ( Highly dependent on the operating system platform )

‍

stay jstack The second line of output is the status of the thread , stay JVM Enumeration is used for thread state in java.lang.Thread.State To express ,State Is defined as follows ：

 public enum State {
        NEW,
        RUNNABLE,
        BLOCKED,
        WAITING,
        TIMED_WAITING,
        TERMINATED;
    }

Copy

In the thread transition diagram ：

jstack Keyword analysis

keyword	explain
deadlock	Deadlock
waiting on condition	Wait for a resource or condition to occur to wake yourself up . Specific needs to be combined with jstacktrace To analyze , For example, the thread is sleep, The network is busy reading and writing and waiting
Blocked	Blocking
waiting on monitor entry	Wait for lock ‍
in Object.wait()	After obtaining the lock, execute obj.wait()

3 solve the problem

3.1 Thread blocking problem

Back to Scene 1 Rely on the above command We print out the simulated jstack Stack

"Quartz-2" #1 prio=5 os_prio=31 tid=0x00007feb6c008000 nid=0x2603 runnable [0x000070000771a000]
   java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:170)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
  at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
  - locked <0x00000007959e1830> (a java.io.BufferedInputStream)
  at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
  at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
  - locked <0x00000007959b1848> (a sun.net.www.protocol.http.HttpURLConnection)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
  - locked <0x00000007959b1848> (a sun.net.www.protocol.http.HttpURLConnection)
  at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
  at java.net.URLConnection.getContentType(URLConnection.java:512)
  at com.courage.platform.sms.client.util.SmsHttpClientUtils.getResponseAsString(SmsHttpClientUtils.java:345)
  at com.courage.platform.sms.client.util.SmsHttpClientUtils.doGet(SmsHttpClientUtils.java:241)

Copy

By the visible on , The current thread is blocking from http Get response data from the link . Maybe the other party httpserver Time consuming , Or the other party server hang Live in the . In this case , I adjusted it properly http client Of connect and read timeout.

3.2 cpu High problem

Back to scene two ,

We use first ps Command to find the pid( If you have several target processes , You can use first top Let's see which occupancy is higher ). Then use

top -H -p pid

Copy

To find the cpu Some threads with high utilization rate （ The following figure is a simulation diagram ）

Then it will take up the highest pid Convert to 16 Base number printf '%x\n' pid obtain nid

And then directly in jstack Find the corresponding stack information in the jstack pid | grep 'nid' -C5 –color

You can see that we have found nid by 0x42 Stack information for .

Live Scores We found the occupation cpu The highest is GC Threads . Then our suspicion is due to GC Caused by high frequency .

And then use jstat every other 1s View memory usage

jstat -gcutil pid 1000    

Copy

It is found that the Cenozoic is basically full every two seconds , And then look at each key Corresponding data size , Discover each value Size is 500k about .

The solution is simple :

• Reduce cache data size
• Appropriately adjust the size of the Cenozoic
• Query the full quantity for the first time , Subsequently passed websocket Push the increment to the front end

4 Coding habits Guide

4.1 Set the thread name

The thread name is set mainly for jstack Easy to query in the stack . stay logback It is better to also mark the name of the offline process in the log . When encountering problems, there is also a query source .

There are two ways to name a thread :

• Set the thread name manually

Thread t = new Thread(new Runnable() {
     @Override
     public void run() {
       //something process 
      }
});
t.setName("mytestThread");
t.start();

Copy

• Thread factory settings （ Source code from rocketmq 4.4）

public class ThreadFactoryImpl implements ThreadFactory {
    private final AtomicLong threadIndex = new AtomicLong(0);
    private final String threadNamePrefix;
    private final boolean daemon;
    public ThreadFactoryImpl(final String threadNamePrefix) {
      this(threadNamePrefix, false);
    }
    public ThreadFactoryImpl(final String threadNamePrefix, boolean daemon) {
      this.threadNamePrefix = threadNamePrefix;
      this.daemon = daemon;
    }
    @Override
    public Thread newThread(Runnable r) {
     Thread thread = new Thread(r, threadNamePrefix + this.threadIndex.incrementAndGet());
      thread.setDaemon(daemon);
      return thread;
    }
}

Copy

Usage mode ：

this.traceExecutor = new ThreadPoolExecutor(
  10, 
  20, 
  1000 * 60, 
  TimeUnit.MILLISECONDS, 
  this.appenderQueue,
  new ThreadFactoryImpl("MQTraceSendThread_"));
traceProducer = getAndCreateTraceProducer(rpcHook);

Copy

4.2 Thread pool isolation

Thread isolation is mainly thread pool isolation , In practice, we will classify the requests , And then it's handled by different thread pools , When there is a problem with the request processing of a service , Does not spread the fault to other thread pools , So that other services are available .

Open source code rocketmq give an example :

Let's see broker Startup time , Send a message & pull news Different thread pools handle requests .

It is divided into

• Send message thread pool
• pull Message thread pool
• Query message thread pool
• Reply message thread pool

The code is as follows :

// Send a message   processor 
this.remotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendProcessor, this.sendMessageExecutor);

//pull news   processor 
this.remotingServer.registerProcessor(RequestCode.PULL_MESSAGE, this.pullMessageProcessor, this.pullMessageExecutor);
this.pullMessageProcessor.registerConsumeMessageHook(consumeMessageHookList);

// Consumption management processor 
ConsumerManageProcessor consumerManageProcessor = new ConsumerManageProcessor(this);
this.remotingServer.registerProcessor(RequestCode.GET_CONSUMER_LIST_BY_GROUP, consumerManageProcessor, this.consumerManageExecutor);

// Transaction processor 
this.remotingServer.registerProcessor(RequestCode.END_TRANSACTION, new EndTransactionProcessor(this), this.endTransactionExecutor);
this.fastRemotingServer.registerProcessor(RequestCode.END_TRANSACTION, new EndTransactionProcessor(this), this.endTransactionExecutor);

Copy

4.3 timeout

In order to avoid large-scale blocking of the route pool , It is very important to set the timeout reasonably . Common timeouts are as follows

• http Read write timeout （jdk httpurlconnection）

cononnection(new URL(n = getCurl), METHOD_POST, ctype);
fillHeaders(conn, headers);
conn.setConnectTimeout(connectTimeout);
conn.setReadTimeout(readTimeout);
out = conn.getOutputStream();

Copy

• Lock time ( In the distributed environment redission)

RLock lock = redisson.getLock("myLock");
// traditional lock method
lock.lock();
// or acquire lock and automatically unlock it after 10 seconds
lock.lock(10, TimeUnit.SECONDS);
// or wait for lock aquisition up to 100 seconds 
// and automatically unlock it after 10 seconds
boolean res = lock.tryLock(100, 10, TimeUnit.SECONDS);
if (res) {
   try {
     ...
   } finally {
       lock.unlock();
   }
}

Copy

• Network communication （ adopt countdownlatch Realization ）

 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
        final long timeoutMillis)
   throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
    final int opaque = request.getOpaque();
    try {
       final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
       this.responseTable.put(opaque, responseFuture);
       final SocketAddress addr = channel.remoteAddress();
       channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
        @Override
        public void operationComplete(ChannelFuture f) throws Exception {
           if (f.isSuccess()) {
              responseFuture.setSendRequestOK(true);
             return;
          } else {
             responseFuture.setSendRequestOK(false);
          }
          responseTable.remove(opaque);
          responseFuture.setCause(f.cause());
          responseFuture.putResponse(null);
           log.warn("send a request command to channel <" + addr + "> failed.");
       }
     });
   RemotingCommand responseCommand =
       responseFuture.waitResponse(timeoutMillis);
       if (null == responseCommand) {
          if (responseFuture.isSendRequestOK()) {
            throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis, responseFuture.getCause());
          } else {
            throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
          }
        }
         return responseCommand;
    } finally {
        this.responseTable.remove(opaque);
     }
}

Copy

rocketmq Encapsulates the responseFuture Object timeout setting passed countdown To achieve the timeout effect

public RemotingCommand waitResponse(final long timeoutMillis) 
throws InterruptedException {
    this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
    return this.responseCommand;
}

Copy

5 Professional product recommendation

5.1. Ali Arthas

Arthas yes Alibaba Open source Java Diagnostic tools . We can see from the figure below With a simple command, you can view the thread when the process is blocked

5.2 PerfMa Stupid horse network

Benma network is committed to building a one-stop IT System stability assurance solution , Focus on performance measurement and tuning 、 The company that locates and solves the root cause of the fault .

Let's see how the community version looks at the stack .

1. First upload dump file
2. From the console Thread pool Stack lock Present at multiple levels jvm Thread information

6 summary

Friends who have played the legend of Jin Yong may have heard “ YQQ ” , This is a very basic move at the beginning of the game , Weak power . Cultivation is also time-consuming , Many players have given up , What I didn't expect , Practice wild ball boxing until 10 level , Players are like getting through Ren Du's two veins , Basically, one hit will kill .

I think learning technology is the same .

1. insist ,never give up Constantly hone your basic skills
2. Dare to practice , When something goes wrong , Don't be scared , This is a once-in-a-lifetime opportunity to learn
3. reflection , Learn from others

Write for the first time , Hopefully that helped .