How does hystrix solve ThreadLocal information loss

This article shares ThreadLocal encounter Hystrix The scheme of time context information transmission .

One 、 background

The author involved in the use of ThreadLocal To store some key information in the context link , Some of these business implementations depend on external interfaces , For these dependent interfaces Hystrix For fuse protection , But in the use of Hystrix It is found in the method of fuse protection to obtain ThreadLocal Information is inconsistent with expectations , This paper aims to explore how to solve this problem .

Two 、ThreadLocal

stay Java In programming languages ThreadLocal Is used to facilitate developers in the same thread context of different classes 、 Sharing information in different ways ,ThreadLocal Variables are not affected by other threads , Different threads are isolated from each other , That is, thread safe . In the actual service link, sometimes some common information needs to be shared from the entrance to the specific service implementation , For example, the unique ID of the user 、 Link tracking unique identification, etc , This information can be used ThreadLocal To store and implement , Here's a simple sharing on the same link traceId Example code for .

public class ThreadLocalUtil {
 
    private static final ThreadLocal<String> TRACE_ID = new ThreadLocal<>();
 
    public static void setTraceId(String traceId) {
        TRACE_ID.set(traceId);
    }
 
    public static String getTraceId() {
        return TRACE_ID.get();
    }
 
    public static void clearTraceId() {
        TRACE_ID.remove();
    }
}

3、 ... and 、Hystrix

In a distributed environment , The external services that each system depends on will inevitably fail or time out ,Hystrix By adding delay fault tolerance and failure tolerance logic to dependent services , It's called 「 Fuse 」, To help developers flexibly control the distributed services they depend on .

Hystrix Access points between services are isolated , Cascading failures between denial of service , And offers the option of downgrading , All this is to provide the overall robustness of the system , In large-scale distributed services , The robustness of the system is particularly important .Hystrix A detailed introduction can be found in ：Hystrix Introduce

Four 、ThreadLocal run into sb. Hystrix

When the specific implementation in the service link depends on external services , And make relevant fuse protection , So the two protagonists of this article meet in this way .

according to Hystrix We know about the related documents of ,Hystrix Provides two thread isolation modes ： Semaphores and thread pools .

In semaphore mode, business logic is executed in the same thread context , The thread pool mode uses Hystrix The thread pool is provided to execute the relevant business logic . In the daily business development, more need to fuse is related to the external network IO Called （ Such as RPC call ）,Hystrix One of the purposes of existence is to reduce the consumption of service container threads by external dependent calls , Semaphore mode is obviously not suitable for , So we use thread pool mode in most scenarios , and Hystrix Thread pool mode is also enabled by default .

What this article wants to solve is the problem that only exists in this default mode ：

1、InheritableThreadLocal

Someone might think if it can be used InheritableThreadLocal To solve the ？

InheritableThreadLocal You can share the thread variable information in the current thread to the one created by the current thread 「 Sub thread 」 in , But there's a very important message missing here ,Hystrix In the thread mode, the bottom layer uses a thread pool maintained by itself , In other words, the threads will be reused , Then it will appear that the information shared by each thread is obtained for the first time before 「 Parent thread 」 Share information for , It's obviously not what we expect , therefore InheritableThreadLocal Excluded .

So you want to be in Hystrix How to solve this problem ？

first-class Hystrix We have provided relevant solutions for you , And it's plug-in , On demand customization .Hystrix For a detailed description of the plug-in, please see this ：Hystrix The plugin is introduced , This paper introduces two kinds of schemes to you .

How to make ThreadLocal Variable information in HystrixCommand Can be executed in Hystrix Correct delivery in threads ？

2、Concurrency Strategy

Use HystrixConcurrencyStrategy Plug ins can be packaged Hystrix The method executed by the thread , Specific direct look at the example code ：

public class MyHystrixConcurrencyStrategy extends HystrixConcurrencyStrategy {
 
    @Override
    public <T> Callable<T> wrapCallable(Callable<T> callable) {
        String traceId = ThreadLocalUtil.getTraceId();
        return () -> {
            ThreadLocalUtil.setTraceId(traceId);
            try {
                return callable.call();
            } finally {
                ThreadLocalUtil.clearTraceId();
            }
        };
    }
}
 
 
//  Register the current policy plug-in somewhere in the business code 
HystrixPlugins.getInstance().registerConcurrencyStrategy(new MyHystrixConcurrencyStrategy());

It's very easy to use this way , As long as the developer will focus on ThreadLocal Value for 「 Copy 」 that will do , That's just the way it is ？

We notice that this approach is essentially aimed at HystrixCommand Of run() Method （ That is to say, we added @HystrixCommand Annotation business approach ） Intercept processing , But it may time out or fail , Then it will be executed fallback Method , If in fallback Methods also want to share relevant context information , This is not the time to cover this scene .

If in your business fallback You don't need to pay attention to the context information , Then the above solution can meet the demand , It's also very simple. . But if fallback Context information is also required in the method , Then you can use Hystrix The following plug-in methods are provided .

3、Command Execution Hook

Use HystrixCommandExecutionHook Can realize the right Hystrix Complete control of the execution process , You can override some of its key node callback methods , To meet your customization needs . For more information, you can see this ：Command Execution Hook Introduce , Here is a list of HystrixCommandExecutionHook Some of the key methods commonly used ：

After understanding these key methods , You can see that the implementation is also very simple , As long as onStart() When 「 Copy 」 Under the context of attention , And then in onExecutionStart() and onFallbackStart() Before the execution of the two methods 「 Paste 」 Under the context of attention , Finally, we are doing the corresponding cleaning up , You can meet your needs , The sample code is as follows ：

public class MyHystrixHook extends HystrixCommandExecutionHook {
     
    private String traceId;
 
    @Override
    public <T> void onStart(HystrixInvokable<T> commandInstance) {
        copyTraceId();
    }
 
    @Override
    public <T> void onExecutionStart(HystrixInvokable<T> commandInstance) {
        pasteTraceId();
    }
 
    @Override
    public <T> void onFallbackStart(HystrixInvokable<T> commandInstance) {
        pasteTraceId();
    }
//  below option1 and option2 Choose one of the overrides 
//------------------------------------option1------------------------------------
    @Override
    public <T> void onExecutionSuccess(HystrixInvokable<T> commandInstance) {
        ThreadLocalUtil.clearTraceId();
        super.onExecutionSuccess(commandInstance);
    }
 
    @Override
    public <T> Exception onExecutionError(HystrixInvokable<T> commandInstance, Exception e) {
        ThreadLocalUtil.clearTraceId();
        return super.onExecutionError(commandInstance, e);
    }
 
    @Override
    public <T> void onFallbackSuccess(HystrixInvokable<T> commandInstance) {
        ThreadLocalUtil.clearTraceId();
        super.onFallbackSuccess(commandInstance);
    }
     
     @Override
    public <T> Exception onFallbackError(HystrixInvokable<T> commandInstance, Exception e) {
        ThreadLocalUtil.clearTraceId();
        return super.onFallbackError(commandInstance, e);
    }
//------------------------------------option1------------------------------------
 
//------------------------------------option2------------------------------------
        @Override
    public <T> void onSuccess(HystrixInvokable<T> commandInstance) {
        ThreadLocalUtil.clearTraceId();
        super.onSuccess(commandInstance);
    }
 
    @Override
    public <T> Exception onError(HystrixInvokable<T> commandInstance, HystrixRuntimeException.FailureType failureType, Exception e) {
        ThreadLocalUtil.clearTraceId();
        return super.onError(commandInstance, failureType, e);
    }
//------------------------------------option2------------------------------------
     
     private void copyTraceId() {
        this.traceId = ThreadLocalUtil.getTraceId();
    }
 
    private void pasteTraceId() {
        ThreadLocalUtil.setTraceId(traceId);
    }
 
}
 
//  Register somewhere in the business code Hook plug-in unit 
HystrixPlugins.getInstance().registerCommandExecutionHook(new MyHystrixHook());

Is this the way to solve the problem ？ Think about it carefully if there is any problem ？

We know HystrixCommandExecutionHook After the plug-in is registered , all HystrixCommand When it is called and executed, it will go through these overriding methods , There will also be multithreaded rewriting traceId, So for this Hook Under the traceId It could be changed at any time . Suppose there's a scenario like this ：

Caller thread 1 Contextual traceId by "t1", In the call to its dependent Hystrix When the method is used ,traceId Is set to "t1"
At the same time, the caller thread 2 Contextual traceId by "t2", In the call to its dependent Hystrix When the method is used , It also triggers changes traceId by "t2"
stay hystrix Threads 1 When you start to execute specific business methods , He wanted to 「 Paste 」 Of traceId It has been changed to "t2", Instead of the initial caller thread 1 Set when "t1"

To solve the problems that we have encountered above ,Hystrix Provides developers with access to HystrixRequestContext and HystrixRequestVariableDefault These two key classes solve .

HystrixRequestContext Used to record every time Hystrix The context of the request , There are two key messages ：
static ThreadLocal<HystrixRequestContext> requestVariables: Used to record every time HystrixCommand Execution context .

ConcurrentHashMap<HystrixRequestVariableDefault<?>, HystrixRequestVariableDefault.LazyInitializer<?>> state： Used to record the real data of the context .

HystrixRequestVariableDefault It's a bit like ThreadLocal, Provides get(),set() Method , The realization of specific ability depends on HystrixRequestContext.

HystrixCommandExecutionHook The example code of the implementation of the ultimate solution of the plug-in is as follows ：


public class MyHystrixHook extends HystrixCommandExecutionHook {
     
    private HystrixRequestVariableDefault<String> requestVariable = new HystrixRequestVariableDefault<>();
 
    public <T> void onStart(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.initializeContext();
                copyTraceId();
    }
 
    @Override
    public <T> void onExecutionStart(HystrixInvokable<T> commandInstance) {
        pasteTraceId();
    }
 
    @Override
    public <T> void onFallbackStart(HystrixInvokable<T> commandInstance) {
        pasteTraceId();
    }
 
        @Override
    public <T> void onSuccess(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        super.onSuccess(commandInstance);
    }
 
    @Override
    public <T> Exception onError(HystrixInvokable<T> commandInstance, HystrixRuntimeException.FailureType failureType, Exception e) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        return super.onError(commandInstance, failureType, e);
    }
     
     private void copyTraceId() {
        requestVariable.set(ThreadLocalUtil.getTraceId());
    }
 
    private void pasteTraceId() {
        ThreadLocalUtil.setTraceId(requestVariable.get());
    }
}

In every time Hook perform onStart() Method time , It has to be executed first HystrixRequestContext Initialization of , Then, we will focus on the context information 「 Copy 」, The key codes are as follows ：

public void set(T value) {
    HystrixRequestContext.getContextForCurrentThread().state.put(this, new LazyInitializer<T>(this, value));
}

Copy the information of interest to a thread related ConcurrentHashMap It's in , According to the front face HystrixCommandExecutionHook We know ,onStart() At the time, the current thread is the caller thread ;

In the real beginning HystrixCommand Business side approach , It needs to be done at this time 「 Paste 」 Context information , from requestVariable.get() obtain ,get The key operation code is as follows ：

public T get() {
      if (HystrixRequestContext.getContextForCurrentThread() == null) {
          throw new IllegalStateException(HystrixRequestContext.class.getSimpleName() + ".initializeContext() must be called at the beginning of each request before RequestVariable functionality can be used.");
      }
      ConcurrentHashMap<HystrixRequestVariableDefault<?>, LazyInitializer<?>> variableMap = HystrixRequestContext.getContextForCurrentThread().state;
     
      // short-circuit the synchronized path below if we already have the value in the ConcurrentHashMap
      LazyInitializer<?> v = variableMap.get(this);
      if (v != null) {
          return (T) v.get();
      }
 
      //  Omit a part of 
      ....
}

As you can see from the code get And set The operation corresponds to , It's also thread related ConcurrentHashMap Get the corresponding value , We also know that the current thread is Hystrix Provided thread pool threads , Not the same thread as the caller thread , Then the context information of business concern can be correctly transmitted to Hystrix In the thread ？ After testing, it does 「 magical 」 The right transmission of , How did you do it ？

Turned out to be Hystrix「 silently 」 Did it for us , Through debugging, we can see the following key code ：

    this.actual = action;
    //  Caller thread HystrixRequestContext Information 
    this.parentThreadState = HystrixRequestContext.getContextForCurrentThread();
 
    this.c = concurrencyStrategy.wrapCallable(new Callable<Void>() {
 
        @Override
        public Void call() throws Exception {
            HystrixRequestContext existingState = HystrixRequestContext.getContextForCurrentThread();
            try {
                //  We did a copy operation for us 
                HystrixRequestContext.setContextOnCurrentThread(parentThreadState);
                //  Start really implementing the business defined approach , At this point, the context information is consistent 
                actual.call();
                return null;
            } finally {
                HystrixRequestContext.setContextOnCurrentThread(existingState);
            }
        }
    });
}

In the execution of business defined HystrixCommand Before the method ,Hystrix The encapsulated object helps us to transfer the context information of the caller thread 「 Copy 」 Over here. , In fact, this processing idea is a bit similar to our previous plug-in HystrixConcurrencyStrategy.

5、 ... and 、 summary

HystrixConcurrencyStrategy and HystrixCommandExecutionHook Both plug-in mode, you can judge according to the actual situation , If you're sure you don't need to be in fallback Pay attention to the context and pass the information , Then use the former , It's also very simple , But if you want to solve it more thoroughly , So the latter way is OK .