Using hystrix to implement fault-tolerant processing of microservices

Use Hystrix Implement fault tolerance of microservice

Preface

By way of the foregoing 《 Use Feign Implement declarative REST call 》https://mp.weixin.qq.com/s/npx_w5Sx0NJDumyhfkohLQ , thus , We've used Eureka It realizes the registration and discovery of micro services , Ribbon The load balancing on the client side is realized ,Feign Realize declarative API call .

This article mainly discusses how to use Hystrix Fault tolerance of microservices .

Means of fault tolerance

If the service provider responds very slowly , Then the request from the consumer to the provider will be forced to wait , Until the provider responds or times out . In high load scenarios , If nothing is done , Such problems may lead to the exhaustion of resources of service consumers or even the collapse of the whole system .

When dependent services are unavailable , Will the service itself be dragged down ？ This is what we should consider .

---

Avalanche effect

The application layer of a microsystem usually contains multiple services . Microservices communicate with each other through the network , So as to support the whole application system , therefore , There is inevitably a dependency relationship between micro services . We know , No micro service is 100% You can use , The Internet is often fragile , Therefore, it is inevitable that some requests will fail .

We often put “ Basic service failure ” Lead to “ Cascading failure ” This phenomenon is called avalanche effect . The avalanche effect describes Provider unavailability causes consumer unavailability , And will not be available for gradual amplification of the process .

---

How to be fault tolerant

To prevent the avalanche effect , There must be a strong fault tolerance mechanism . The fault tolerance mechanism needs to realize the following two points ：

• Set timeout for network requests
• Use breaker mode
  • If a request for a microservice has a large number of timeouts （ It often indicates that the microservice is unavailable ）, There is no point in letting new requests access the service , It will only unnecessarily consume resources . for example , The timeout time is set to 1s , If there are a large number of requests in a short time, they can't be in 1s Get a response inside , There is no need to request dependent services .
  • The circuit breaker can be understood as an agent for operation that is easy to cause errors . This proxy can count the number of failed calls over a period of time , And decide whether to request the dependent service normally or return directly .
  • The circuit breaker can achieve fast failure , If it detects many similar errors over a period of time （ For example, overtime ）, It will be in a period of time , Forcing calls to the service to fail quickly , That is, no longer requesting the services on which it depends . such , Applications don't have to waste CPU Time to wait for a long timeout .
  • The circuit breaker can also automatically diagnose whether the dependent service has returned to normal . If it is found that the dependent service has returned to normal , Then the request for the service will be resumed . In this way , You can implement the “ Self repair ” – When the dependent service is abnormal , Quick failure when opening circuit breaker , So as to prevent avalanche effect ; When the dependent service is found to be back to normal , The request will be resumed .

Logic of circuit breaker state transition , As shown in the figure below ：

• Under normal circumstances , The circuit breaker is closed , You can normally request dependent services .
• When for a while , The request failure rate reaches a certain threshold （ For example, the error rate reaches 50% , or 100 Time / Minutes, etc ）, The circuit breaker will open here , No more requests for dependent services .
• After the circuit breaker is opened for a period of time , Will automatically enter “ half ” state . here , Circuit breakers allow a request to access dependent services . If the request can be called successfully , Turn off the circuit breaker ; Otherwise, keep it open .

Hystrix brief introduction

Hystrix A tool class library that implements timeout mechanism and circuit breaker pattern .

Hystrix By Netflix An open source delay and fault tolerance Library , Used to isolate access to remote systems 、 Services or third-party libraries , Prevent cascading failures , So as to improve the availability and fault tolerance of the system .

Hystrix Mainly through the following points to achieve delay and fault tolerance ：

• Package request ： Use HystrixCommand （ or HystrixObservableCommand ） Package call logic for dependencies , Each command is executed in a separate thread . This uses the “ Command mode ”.
• Trip mechanism ： When the error rate of a service exceeds a certain threshold ,Hystrix It can trip automatically or manually , Stop requesting the service for a period of time .
• Resource isolation ：Hystrix A small thread pool is maintained for each dependency （ Or semaphores ） . If the thread pool is full , Requests to that dependency are rejected immediately , Instead of waiting in line , So as to speed up the failure determination .
• monitor ：Hystrix It can monitor the changes of operation index and configuration in near real time , For example, success 、 Failure 、 Timeout and rejected requests, etc .
• Fallback mechanism ： When the request fails 、 Overtime 、 Be rejected , Or when the circuit breaker is on , Perform fallback logic . The fallback logic can be provided by the developer , For example, return a default value .
• Self repair ：： After the circuit breaker is opened for a period of time , Will automatically enter “ half ” state .

Hystrix Thread isolation strategy

execution.isolation.strategy

Hystrix There are two isolation strategies for ： Thread isolation and semaphore isolation .

• THREAD（ Thread isolation ）： Use this method , HystrixCommand Will execute on a separate thread , Concurrent requests are limited by the number of threads in the thread pool .
• SEMAPHORE （ Semaphore isolation ）： Use this method ,HystrixCommand Will execute on the calling thread , The cost is relatively small , Concurrent requests are limited by the number of semaphores .

Hystrix Thread isolation is default and recommended in （THREAD） , Because this method has an additional layer of protection besides network timeout .

Generally speaking , Only when the invocation load is very high （ For example, each instance is called hundreds of times per second ） You need semaphore isolation , Because in this scenario THREAD The cost will be higher . Semaphore isolation is generally only applicable to the isolation of non network calls .

You can use execution.isolation.strategy Property specifies the isolation policy .

    @HystrixCommand(fallbackMethod = "findByIdFallback", commandProperties = {
    
        @HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE")
    })
    @GetMapping("/{id}")
    public User findById(@PathVariable Long id){
    
        //...
    }

Summary ：

• Hystrix The isolation strategies for are THREAD and SEMAPHORE Two kinds of , The default is THREAD.
• Under normal circumstances , Keep the default .
• If a runtime exception occurs where the context cannot be found , Consider setting the isolation policy to SEMAPHORE .

Integrate in a common way Hystrix

1. Copy project micro-consumer-movie-ribbon , take artifactId It is amended as follows micro-consumer-movie-ribbon-hystrix .

2. Add dependency .

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>

3. Add comments to the startup class @EnableCircuitBreaker , Enable breaker support for the project .

@SpringBootApplication
@EnableDiscoveryClient
@EnableCircuitBreaker
public class MicroConsumerMovieHystrixApplication {
    
    public static void main(String[] args) {
    
        SpringApplication.run(MicroConsumerMovieHystrixApplication.class, args);
    }
}

4. modify MovieController , Let one of them findById Method has fault tolerance .

    @HystrixCommand(fallbackMethod = "findByIdFallback")
    @GetMapping("/{id}")
    public User findById(@PathVariable Long id){
    
        User entity = restTemplate.getForObject("http://micro-provider-user/user/v1/"+id, User.class);
        return entity;
    }

    public User findByIdFallback(Long id){
    
        User user = new User();
        user.setId(-1L);
        user.setName(" The default user ");
        return user;
    }

by findById() Method writes a fallback method findByIdFallback() The method and findById() Method has the same parameter and return value type , This method returns a default User .

stay findById() On the way , Using annotations @HystrixCommand(fallbackMethod = "findByIdFallback") fallbackMethod attribute , Specify the fallback method findByIdFallback() .

5. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-ribbon-hystrix ; visit ：http://localhost:8010/movie/v1/1 , Normal return result .

6. Stop the service provider ：micro-provider-user , Revisit ：http://localhost:8010/movie/v1/1 , Returns the default result in the fallback method .

When the request fails 、 Be rejected 、 Timeout or when the circuit breaker is opened , Will enter the fallback method . But entering the fallback method does not mean that the circuit breaker has been opened .

If you need to obtain the information that leads to fallback Why , Just in fallback Method Throwable Parameters can be ：

public User findByIdFallback(Long id, Throwable throwable){
    
    logger.error("error:",throwable);
    User user = new User();
    user.setId(-1L);
    user.setName(" The default user ");
    return user;
}

In most cases , When a business exception occurs , We don't want to trigger fallback What to do at this time ？ Hystrix There is one HystixBadRequestException class , This is a special exception class , When the exception occurs , Does not trigger fallback . therefore , Custom business exceptions can be inherited from this class , So as to achieve the effect that business exceptions do not return .

in addition ,HystrixCommand For us ignoreExceptions attribute , You can also use this attribute to configure exception classes that do not want to perform fallback . example ：
@HystrixCommand(fallbackMethod = "findByIdFallback", ignoreExceptions={IllegalArgumentException.class, MyBusinessException.class}) .

Feign Use Hystrix

Spring Cloud The default is Feign Integrated Hystrix .

Spring Cloud Dalston In the previous version ,Feign The default is on Hystrix Support , No need to set feign.Hystrix.enabled=true .

Spring Cloud Dalston Start , Feign Of Hystrix Support , Off by default , You have to set feign.Hystrix.enabled=true attribute .

1. Create project . Copy project micro-consumer-movie-feign , take artifactId It is amended as follows micro-consumer-movie-feign-hystrix-fallback .

2. modify Feign Interface .

@FeignClient(name = "micro-provider-user", fallback = UserFeignClientFallback.class)
public interface UserFeignClient {
    

    @RequestMapping(value = "/user/v1/{id}", method = RequestMethod.GET)
    User findById(@PathVariable("id") Long id);
}

@Component
public class UserFeignClientFallback implements UserFeignClient {
    
    @Override
    public User findById(Long id) {
    
        User user = new User();
        user.setId(-1L);
        user.setUsername(" The default user ");
        return user;
    }
}

Just use @FeignClient Annotated fallback attribute , You can specify a name for Feign Client add fallback .

3. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-feign-hystrix-fallback ; visit ：http://localhost:8010/movie/v1/1 , Normal return result .

4. Stop the service provider ：micro-provider-user , Revisit ：http://localhost:8010/movie/v1/1 , Returns the default result in the fallback method .

---

about Feign , How to get the reason for fallback ？ You can use annotations @FeignClient Of fallbackFactory attribute .

1. Create project . Copy project micro-consumer-movie-feign , take artifactId It is amended as follows micro-consumer-movie-feign-hystrix-fallback-factory .

2. modify Feign Interface .

@FeignClient(name = "micro-provider-user", fallbackFactory = UserFeignClientFallbackFactory.class)
public interface UserFeignClient {
    
    @RequestMapping(value = "/user/v1/{id}", method = RequestMethod.GET)
    User findById(@PathVariable("id") Long id);
}

@Component
public class UserFeignClientFallbackFactory implements FallbackFactory<UserFeignClient> {
    
    private static final Logger logger = LoggerFactory.getLogger(UserFeignClientFallbackFactory.class);

    @Override
    public UserFeignClient create(Throwable throwable) {
    
        return new UserFeignClient() {
    
            @Override
            public User findById(Long id) {
    
                //  It's best to put the log in each  fallback  In the method , Don't put it directly on  create  In the method , Otherwise, when the application starts , The log will be printed 
                logger.info("fallback; reason was:", throwable);
                User user = new User();
                user.setId(-1L);
                user.setUsername(" The default user ");
                return user;
            }
        };
    }
}

3. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-feign-hystrix-fallback-factory ; visit ：http://localhost:8010/movie/v1/1 , Normal return result .

4. Stop the service provider ：micro-provider-user , Revisit ：http://localhost:8010/movie/v1/1 , Returns the default result in the fallback method .

Hystrix Monitoring of

In addition to achieving fault tolerance ,Hystrix It also provides near real-time monitoring .

HystrixCommand and HystrixObservableCommand When executed , It will generate execution results and running indicators , For example, the number of requests executed per second 、 Number of successes, etc , These monitoring data are very useful for analyzing the state of the application system .

Use Hystrix Module hystrix-metrics-event-stream , These monitored indicator information can be displayed in text/event-stream The format is exposed to external systems .spring-cloud-starter-hystrix This module is already included in .

Add... To the project spring-boot-starter-actuator , You can use /hystrix.stream Endpoint acquisition Hystrix Monitoring information for .

1. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-ribbon-hystrix ; visit ：http://localhost:8010/hystrix.stream , You can see that the browser is always in the requested State , Page blank . This is because the project has been annotated at this time @HystrixCommand The method of has not been implemented yet , So there is no monitoring data .

2. visit http://localhost:8010/movie/v1/1 after , Revisit ：http://localhost:8010/hystrix.stream , You can see that the monitoring data will continue to appear on the page .

This is because the system will constantly refresh to obtain real-time monitoring data . Hystrix The monitoring indicators are very comprehensive , for example HystrixCommand The name of 、 group name 、 Circuit breaker status 、 Error rate 、 Number of errors, etc .

---

Feign Project Hystrix monitor

1. Create project . Copy project micro-consumer-movie-feign-hystrix-fallback , take artifactId It is amended as follows micro-consumer-movie-feign-hystrix-fallback-stream .

2. Add dependencies to the project ：

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

3. Add on the startup class @EnableCircuitBreaker , It can be used in this way /hystrix.stream End point .

4. Modify the startup port number , Subsequent aggregation of multiple /hystrix.stream End point .

server:
  port: 8020

5. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-feign-hystrix-fallback-stream .

6. visit http://localhost:8020/movie/v1/1 after , Revisit ：http://localhost:8020/hystrix.stream , You can see that the monitoring data will continue to appear on the page .

Use Hystrix Dashboard Visual monitoring data

visit /hystrix.stream The data obtained by the endpoint is displayed in text . It's hard to get through these data , At a glance, you can see the current operation state of the system .

You can use Hystrix Dashboard , Make monitoring data graphical 、 visualization .

Let's write a Hystrix Dashboard .

1. Use Spring Boot Build projects quickly micro-hystrix-dashboard .

2. Add dependencies to the project ：

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
</dependency>

3. Add comments to the startup class @EnableHystrixDashboard.

@SpringBootApplication
@EnableHystrixDashboard
public class HystrixDashboardApplication {
    
    public static void main(String[] args) {
    
        SpringApplication.run(HystrixDashboardApplication.class, args);
    }
}

4. Set the service startup port number .

server:
  port: 8030

5. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-ribbon-hystrix 、Hystrix Dashboard ：micro-hystrix-dashboard .

6. visit http://localhost:8010/movie/v1/1 after , Revisit ：http://localhost:8010/hystrix.stream , You can see that the monitoring data will continue to appear on the page .

7. visit http://localhost:8030/hystrix , And enter the http://localhost:8010/hystrix.stream , Click on “Monitor Stream" You can see the visual monitoring data .

The circle in the upper left corner represents the flow and status of the method ：

• The larger the circle, the greater the method traffic ;
• The green circle represents the health of the circuit breaker 、 Yellow represents an accidental failure of the circuit breaker 、 Red represents circuit breaker failure ;

The counter in the upper right corner （ Three columns of numbers ）：

The first column goes from top to bottom ：

• Green represents the current number of successful calls
• Blue represents the number of short circuit requests
• Cyan represents the number of bad requests

The second column goes from top to bottom ：

• Yellow represents the number of timeout requests
• Purple represents the number of thread pool rejections
• Red represents the number of failed requests

The third column ：

• In the past 10s Percentage of bad requests

Thread Pools：

• Hystrix A corresponding thread pool will be created for a protected class , The purpose of this is Hystrix When your command is called , Not affected by the method request thread （ Or say Hystrix The worker thread and the caller thread do not affect each other ）.

The circle in the upper left corner represents the flow and status of the thread pool ：

• The larger the circle, the more active the thread pool is , The more traffic
• The circle color represents the health of the thread pool

Bottom left corner from top to bottom ：

• Active Represents the number of active threads in the thread pool
• Queued Represents the number of threads queued , This function is disabled by default , Therefore, it is always... By default 0
• Pool Size Represents the number of threads in the thread pool

Bottom right corner from top to bottom ：

• Max Active Represents the largest active thread , The data shown here is in the current adoption cycle , Maximum number of active threads
• Execcutions Represents that the thread in the thread pool is called to execute Hystrix The number of orders
• Queue Size Represents the size of the thread pool queue , Default disabled , meaningless

Hystrix Dashboard See... For parameter description ：https://blog.csdn.net/qq_41125219/article/details/121370276

Try to set the isolation policy to SEMAPHORE , At this point, the ThreadPool The column will no longer show . This is because THREAD and SEMAPHORE Different implementation mechanisms of .

Use Turbine Aggregate monitoring data

The application system using microservice architecture usually contains several microservices , Each microservice typically deploys multiple instances . If you can only view the monitoring data of a single instance at a time , Must be in Hystrix Dashboard Switch the address you want to monitor , It's obviously inconvenient . How to solve this problem ？

Turbine brief introduction

Turbine It's a convergence Hystrix Tools for monitoring data , It can put all relevant Hystrix.stream The data of the endpoint is aggregated into a combined turbine.stream in , Thus, the monitoring of the cluster is more convenient .

1. Use Spring Boot Build projects quickly micro-hystrix-turbine .

2. Add dependencies to the project ：

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-turbine</artifactId>
</dependency>

3. Add comments to the startup class @EnableTurbine.

@SpringBootApplication
@EnableTurbine
public class TurbineApplication {
    
    public static void main(String[] args) {
    
        SpringApplication.run(TurbineApplication.class, args);
    }
}

4. Modify the configuration file

server:
  port: 8031

spring:
  application:
    name: micro-hystrix-turbine

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
  instance:
    prefer-ip-address: true

turbine:
  appConfig: micro-consumer-movie,micro-consumer-movie-feign-hystrix-fallback-stream
  clusterNameExpression: "'default'"

Turbine Will be in Eureka Server Find micro-consumer-movie,micro-consumer-movie-feign-hystrix-fallback-stream These two microservices , And aggregate the monitoring data of two microservices .

5. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-ribbon-hystrix and micro-consumer-movie-feign-hystrix-fallback-stream、Hystrix Dashboard ：micro-hystrix-dashboard 、Turbine ：micro-hystrix-turbine. visit http://localhost:8031/turbine.stream .

6. visit http://localhost:8010/movie/v1/1 and http://localhost:8020/movie/v1/1 , Revisit ： http://localhost:8031/turbine.strea.

7. visit http://localhost:8030/hystrix , And enter the http://localhost:8031/turbine.stream , Click on “Monitor Stream" You can see the visual monitoring data .

Use Rabbitmq Message oriented middleware collects monitoring data

In some cases , For example, microservices and Turbine The Internet is not working , here , With the help of message oriented middleware, data collection can be realized. Each micro service will Hystrix Command The monitoring data is sent to the message middleware ,Turbine Consume data in message oriented middleware .

1. use Docker + Docker-Composer Quickly deploy a on a virtual machine Rabbitmq .

2. Create project . Copy micro-consumer-movie-ribbon-hystrix , modify artifactId by micro-consumer-movie-ribbon-hystrix-turbine-mq .

3. Add dependencies to the project ：

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-netflix-hystrix-stream</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

4. Modify the configuration file

server:
  port: 8010

spring:
  application:
    name: micro-consumer-movie
  rabbitmq:
    host: 192.168.233.131
    port: 5672
    username: root
    password: Pwd
    virtual-host: vhost

eureka:
  client:
    service-url:
      default-zone: http://localhost:8761/eureka/
  instance:
    prefer-ip-address: true

---

reform Turbine

1. Create project . Copy project micro-hystrix-turbine , modify artifactId by micro-hystrix-turbine-mq .

2. Add dependencies to the project ：

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-turbine-stream</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

Delete spring-cloud-starter-turbine rely on .

3. Modify the startup class , adding annotations @EnableTurbineStream

@SpringBootApplication
@EnableTurbineStream
public class TurbineMqApplication {
    
    public static void main(String[] args) {
    
        SpringApplication.run(TurbineMqApplication.class, args);
    }
}

4. Modify the configuration file

server:
  port: 8031

spring:
  application:
    name: micro-hystrix-turbine
  rabbitmq:
    host: 192.168.233.131
    port: 5672
    username: root
    password: Pwd
    virtual-host: vhost

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
  instance:
    prefer-ip-address: true

Delete turbine.appConfig and turbine.clusterNameExpression To configure .

5. Start the test . start-up Eureka Server： micro-discovery-eureka 、 Service providers ：micro-provider-user 、 Serving consumers ：micro-consumer-movie-ribbon-hystrix-turbine-mq、Hystrix Dashboard ：micro-hystrix-dashboard 、Turbine ：micro-hystrix-turbine-mq. visit http://localhost:8031/turbine.stream .

6. visit http://localhost:8010/movie/v1/1 , Revisit ： http://localhost:8031/turbine.strea.

7. visit http://localhost:8030/hystrix , And enter the http://localhost:8031/turbine.stream , Click on “Monitor Stream" You can see the visual monitoring data .

Code warehouse

https://gitee.com/chentian114/spring-cloud-practice

official account

Reference resources

《Spring Cloud And Docker Microservice architecture practice 》 Zhou Li