I think it is necessary to write a general idempotent component

Contents of this article

background
Simple idempotent implementation

2.1 Database record judgment

2.2 Concurrency problem solving

Universal idempotent implementation

3.1 design scheme

3.1.1 Universal storage

3.1.2 Easy to use

3.1.3 Support annotation

3.1.4 Multilevel storage

3.1.5 Concurrent reading and writing

3.1.6 Execute the process

3.2 Idempotent interface

3.3 Idempotent annotation

3.4 Automatically distinguish duplicate requests

3.5 Storage structure

3.6 Source code address

background

Answer questions from friends ： Is there any general scheme and Practice for idempotent ？

About what is idempotent , This article will not elaborate . I'm sure you all know that , And have encountered similar problems and have their own set of solutions .

Basically, idempotency in all business systems is handled separately , It's not to say that it can't be dealt with in a unified way , There will be a lot to consider if we deal with it in a unified way .

I personally think that the core business is suitable for the business side to handle by itself , For example, order payment , There's going to be a payment record , An order can only be paid once , The idempotent effect can be achieved through the payment record form .

There are other businesses that are not core , But there is also a need for idempotency . For example, network problems , Multiple retries . The user clicks many times and so on . In this scenario, we still need a general idempotent framework to deal with , Will make business development easier .

Simple idempotent implementation

The realization of idempotent is not complicated , There are many kinds of solutions , First of all, it introduces the scheme based on database record to realize , The general scheme will be introduced later .

Database record judgment

Take the payment scenario at the beginning of the article as an example . The business scenario is that an order can only be paid once , So we will judge whether the order has been paid before payment , If not, pay , If you have paid , The payment was successful , idempotent .

This requires an extra table to store the actions done , In order to judge whether it has been done before .

It's like you're old , And then it's a single tech house . Your family is in a hurry at this time , Your mother introduces you to your little sister every day . You have to dress up every weekend , To meet the little sister your mother introduced you to .

You have to record it before you go ,8 I met in the first week of the month XXX, I saw you the next week YYY, If you're asked to meet again in the third week XXX, If you don't like XXX, You'll take a look at your little book , I've seen this before , There's no need to see you again , Otherwise, it would be embarrassing to meet you .

Concurrency problem solving

By querying payment records , There is no problem in business logic to determine whether payment can be made . But there are problems in concurrent scenarios .

1001 Two payment requests were initiated for the order of , The current two requests query the payment record at the same time , We didn't find out , And then they all started to go through the logic of payment , Finally, we found that the same order was paid twice , This is the idempotent problem caused by concurrency .

There are also many solutions to concurrency , Simple point of the direct use of the database's unique index to solve , A little troublesome will use distributed locks to lock the same resource .

For example, we have orders for 1001 To lock , If two payment requests are initiated at the same time , Then only one request can acquire the lock at a time , Another request that cannot obtain the lock can fail directly , You can also wait for the previous request to complete .

If you wait for the previous request to complete , And then deal with it , You can find out 1001 It has been paid for , Direct return payment succeeded .

Universal idempotent implementation

In order to enable us to focus more on the development of business functions , Idempotent operations in simple scenarios can be handled by unified encapsulation , Let's introduce the implementation of general idempotent .

design scheme

Universal storage

Generally, if we do idempotent inside the program, we query first , Then according to the query results do the corresponding operation . At the same time, the same resources will be locked to avoid concurrency problems .

Locking is universal , The unusual part is to judge whether the operation has been operated before , So we need to have a common store to record all operations .

Easy to use

Provides general idempotent components , The idempotent can be realized by injecting the corresponding class , Shielding and locking , Record, judge, etc .

Support annotation

Except for idempotent control through code , And to make it easier to use , You also need to provide annotations to support idempotency , Users only need to add corresponding annotations on the corresponding business methods , In this way, idempotent can be realized .

Multilevel storage

Need to support multi-level storage , For example, primary storage can use Redis To achieve , The advantage is high performance , Apply to 90% Scene . Because many scenarios are designed to prevent problems caused by repeated requests in a short period of time , By setting a certain failure time , Give Way Key Automatic failure .

Secondary storage can support Mysql, Mongo Such as the database , It is suitable for scenes with long time or permanent storage .

You can specify what primary storage uses by configuration , What does secondary storage use . This scenario is well suited to be implemented with a policy pattern .

Concurrent reading and writing

The introduction of multi-level storage is bound to involve concurrent read-write scenarios , There are two ways to support , Sequence and concurrency .

Order is to write first level storage , Write secondary storage , It's the same with reading . The problem is that the performance will be a little bit wasted .

Concurrency is multithreading writing at the same time , Read at the same time , Improve performance .

Idempotent execution process

Idempotent interface

Idempotent interface definition

  
   public interface DistributedIdempotent {
       /**
        *  Idempotent execution 
        * @param key  idempotent Key
        * @param lockExpireTime  Expiration time of lock 
        * @param firstLevelExpireTime  Primary storage expiration time 
        * @param secondLevelExpireTime  Secondary storage expiration time 
        * @param timeUnit  Storage time unit 
        * @param readWriteType  Reading and writing types 
        * @param execute  The logic to execute 
        * @param fail Key Already exist , Execution logic after idempotent interception 
        * @return
        */
       <T> T execute(String key, int lockExpireTime, int firstLevelExpireTime, int secondLevelExpireTime, TimeUnit timeUnit, ReadWriteTypeEnum readWriteType, Supplier<T> execute, Supplier<T> fail);
   }

Usage mode

  
   /**
    *  Code mode idempotent - There is a return value 
    * @param key
    * @return
    */
   public String idempotentCode(String key) {
       return distributedIdempotent.execute(key, 10, 10, 50, TimeUnit.SECONDS, ReadWriteTypeEnum.ORDER, () -> {
           System.out.println(" Came in ....");
           return "success";
       }, () -> {
           System.out.println(" repeated ....");
           return "fail";
       });
   }

Idempotent annotation

Using annotations , It makes it easier to use , For example, our transaction processing , Caching and so on all use annotations to simplify logic .

Idempotent scenarios can also define general annotations to simplify the use of difficulty , Add comments on business methods that need to support idempotency , Configure basic information .

idempotentHandler It is the method to execute after triggering idempotent rules , That's when we use code to implement idempotent Supplier fail Parameters . It's implemented with Ali Sentinel Current limiting , The processing logic after fusing .

In an idempotent situation , If it's repetitive , Usually return the same result as normal execution .

  
   /**
    *  The way of annotation is idempotent - Specifies the method to be executed after the idempotent rule is triggered 
    * @param key
    */
   @Idempotent(spelKey = "#key", idempotentHandler = "idempotentHandler", readWriteType = ReadWriteTypeEnum.PARALLEL, secondLevelExpireTime = 60)
   public void idempotent(String key) {
       System.out.println(" Came in ....");
   }
   public void idempotentHandler(String key, IdempotentException e) {
       System.out.println(key + ":idempotentHandler It has been carried out ....");
   }

Automatically distinguish duplicate requests

Code handling idempotent , Need to pass in idempotent Key, The annotation method deals with idempotency , The supporting configuration Key, Support SPEL expression . Both of them need to be used to determine the uniqueness of idempotency .

There is also a more common scenario of idempotency , It is to prevent repeated submission or network problems, timeout and try again . The same operation will be requested many times , In this scenario, you can apply for a unique ID, Bring it to the back end every time it's requested , This will identify the uniqueness of the entire request .

I currently have a function that automatically generates a unique identifier , In short, it's based on the requested information MD5, If MD5 No change in value is considered the same request .

Need to carry out MD5 There is a request for the content of URL Parameters , Request body , Request header information . The information in the request header is not related to the specified user Key The scene will be all stitched together , If the request header is configured userId For the user's logo , Then you can only use userId.

Idempotent at the entry of the request Key Automatic generation of , If you do not specify when using idempotent annotations spelKey, You'll use auto generated Key.

Storage structure

Redis: Use String Type storage ,Key It's idempotent Key, Value The default is 1.

Mysql: You need to create a record table .（ Expired data needs to be cleaned up regularly , It can also be stored permanently ）

  
   CREATE TABLE `idempotent_record` (
     `id` int(11) NOT NULL AUTO_INCREMENT COMMENT ' Primary key ',
     `key` varchar(50) NULL DEFAULT '',
     `value` varchar(50) NOT NULL DEFAULT '',
     `expireTime` timestamp NOT NULL COMMENT ' Expiration time ',
     `addTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT ' Creation time ',
     PRIMARY KEY (`id`)
   ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT=' Idempotent records ';

Mongo: Fields follow Mysql equally , convert to Json The format is just .Mongo The collection is automatically created .

About author ： Yin Jihuan , Simple technology enthusiasts ,《Spring Cloud Microservices - Full stack technology and case analysis 》, 《Spring Cloud Microservices introduction Actual combat and advanced 》 author , official account Ape world Originator .

I have compiled a complete set of learning materials , Those who are interested can search through wechat 「 Ape world 」, Reply key 「 Learning materials 」 Get what I've sorted out Spring Cloud,Spring Cloud Alibaba,Sharding-JDBC Sub database and sub table , Task scheduling framework XXL-JOB,MongoDB, Reptiles and other related information .