Deadlock event of thread pool

Everyone pays attention to official account , More high-quality tutorial series are in the centrifuge program ：

 

|  background

Here's the thing , A project in hand uses thread pools to handle a large number of asynchronous tasks , The initial goal was to reduce time-consuming for parallel processing , But the traffic the day before yesterday was offline due to other colleagues job It has increased dozens of times , Let's not discuss this normative problem first . Due to flow spikes , The service timed out , Cause upstream services to fail to get results , The timeout configuration of the front end for their server is unreasonable , It directly leads to white pages , That sounds ridiculous

, The general problem request process is as follows ：

|  screening

Timeout occurs but the service CPU Occupation is not high , There is no trigger HPA（ Dynamic capacity ）, We checked the threads of a machine in the cluster. A large number of threads TimeWait, And locate the problem code , First, let's look at our thread pool definition （ Example ）​​​​​​​

ThreadPoolExecutor tpe = new ThreadPoolExecutor(coreNum,coreNum*2,                100L, TimeUnit.MILLISECONDS,                new LinkedBlockingQueue<>(),namedThreadFactory);

Questions as follows ：

• There is no limit to the length of the queue

• It is better to add logs or alarms when the rejection strategy goes to

Then the most critical place , There is such a parent-child task relationship in this project ：​​​​​​​

/**
     *  Father task , It executes subtasks asynchronously 
     */
    static class FatherTaskTwo implements Callable<String> {
        @Override
        public String call() throws Exception {
            SonTask sonTask = new SonTask();
            Future<String> future = tpe.submit(sonTask);
            return future.get();
        }
    }
    /**
     *  The subtasks 
     */
    static class SonTask implements Callable<String> {
        @Override
        public String call() throws Exception {
            // Deal with some business logic 
            return null;
        }
    }
    public static void main(String[] args) throws Exception {
        FatherTaskTwo fatherTaskTwo = new FatherTaskTwo();
        Future<String> future = tpe.submit(fatherTaskTwo);
        // Main thread adds timeout get Blocking 
        future.get(1000L, TimeUnit.MILLISECONDS);
    }

Here are a few questions ：
1. Father task get The result of the subtask has no timeout

2. Parent and child tasks share a thread pool

What's the problem with this ？ Is the key to today ： Deadlock

We assume that there are at most four threads processing at the same time , Our normal situation may be like this ：

ok, Why is this normal , You can imagine sub tasks 1 And subtasks 2 Will be quickly implemented , So the subtask 3 and 4 Will be executed immediately , But the parent task 3 And parent task 4 Is to wait until the subtask 3 And subtasks 4 The result will be returned after execution , Because father and son are dependent , And in the parent task get Subtask is   No timeout set , But the situation is good , Because subtasks 34 In subtask 12 After execution, you can get the thread resources to execute , Then put the parent task 34 Also digest , But if this is the case ：

Then there are four parent tasks that cannot be digested , The reason lies in ：

1. The parent task depends on its own child task

2. The parent task has no timeout , It will keep blocking , Conditions that cannot break the deadlock

This creates a deadlock , The thread is suspended due to blocking, so CPU It will not rise due to the problem of thread pool

|  Conclusion

Although the whole event link has many unreasonable places , For example, degradation between services 、Online/Offline And so on , But we mainly focus on the internal problems of the application , It is mainly the deadlock caused by the wrong use of thread pool , Then there are roughly several solutions from different angles ：

1. Thread pool isolation , Dependent threads process in different pools

2. Be sure to set get Timeout time

3. There is no need to dismantle so many levels of tasks , Try to be as flat as possible