[technical dry goods] how to ensure the idempotency of the interface?

Hello everyone , I meet you again , I'm the king of the whole stack .

Preface

Interface idempotence problem , For developers , It's a public issue that has nothing to do with language . This article shares some practical solutions to these problems , Most of the content I have practiced in the project , A reference for those in need .

I don't know if you've ever come across these scenes ：

1. Sometimes we're filling in certain form Forms when , The Save button was accidentally pressed twice , There are two duplicate data in the table , It's just id Dissimilarity .
2. We are in the project to solve Interface timeout problem , It's usually introduced Retry mechanism . The first request interface timed out , The requester failed to get the returned result in time （ At this point, it may have been successful ）, To avoid returning incorrect results （ In this case, it is impossible to return to failure directly ？）, The request is then retried several times , This also creates duplicate data .
3. mq When a consumer reads a message , Sometimes it reads Repeat the message （ As for the reason, let's not say , Interested partners , You can talk to me in private ）, If not handled well , It also produces duplicate data .

you 're right , These are all idempotent problems .

Interface idempotence The result of one request or multiple requests is the result of the same operation , It won't have side effects due to multiple clicks .

This kind of problem often occurs in the interface ：

• insert operation , In this case, multiple requests , There may be duplicate data .
• update operation , If you just update the data , such as ：update user set status=1 where id=1, There is no problem . If there's a calculation , such as ：update user set status=status+1 where id=1, In this case, multiple requests , May cause data errors .

So how do we guarantee the idempotency of interfaces ？ This article will tell you the answer .

1. insert before select

Usually , In the interface where the data is stored , We want to prevent duplicate data , Usually in insert front , First, according to name or code Field select Here's the data . If the data already exists , execute update operation , If it doesn't exist , To perform insert operation .

This solution may be when we are usually preventing duplicate data , The most used solution . But this scheme is not suitable for concurrent scenarios , In a concurrent scenario , It should be used together with other schemes , Otherwise, duplicate data will also be generated . I'll mention it here , It's to avoid people stepping on the pit .

2. Add pessimism lock

In the payment scenario , user A My account balance is 150 element , I want to transfer out 100 element , Under normal circumstances, users A All that's left is 50 element . In general ,sql That's true ：

update user amount = amount-100 where id=123;

Copy

If the same request appears more than once , May cause users to A The balance becomes negative . This situation , user A I'm going to cry . At the same time , System developers may also cry , Because it's a very serious system bug.

To solve this problem , You can put a pessimistic lock on it , Will the user A That line of data is locked , Only one request is allowed to acquire a lock at the same time , Update data , Other requests wait .

In general, it is through the following sql Lock single line data ：

select * from user id=123 for update;

Copy

The specific process is as follows ：

Specific steps ：

1. Multiple requests based on id Query user information .
2. Judge whether the balance is insufficient 100, If the balance is insufficient , Then return the insufficient balance directly .
3. If the balance is sufficient , Through for update Query user information again , And try to get the lock .
4. Only the first request can get the row lock , The rest have no requests to acquire locks , Then wait for the next chance to acquire the lock .
5. After the first request gets the lock , Judge whether the balance is insufficient 100, If the balance is enough , Is to update operation .
6. If the balance is insufficient , The description is a repeat request , Then directly return to success .

Here's the thing to watch out for ： If you are using mysql database , The storage engine must use innodb, Because it supports transactions . Besides , here id Fields must be primary keys or unique indexes , Or it'll lock the whole watch .

Pessimistic locks need to lock a row of data during the same transaction operation , If the transaction takes a long time , There will be a lot of requests waiting , Affect interface performance .

Besides , It's hard for the interface to guarantee the same return value for every request , So it's not suitable for idempotent design scenarios , But it can be used in anti heavy scenes .

By the way, here , Anti weight design and Idempotent design , There is a difference . Anti duplication design is mainly to avoid duplicate data , There are not many requirements for interface return . Idempotent design can avoid duplicate data , It also requires that each request return the same result .

3. Lock in optimism

Since pessimistic locks have performance problems , To improve interface performance , We can use the optimistic lock . You need to add a timestamp perhaps version Field , Here we use version Field as an example .

Query the data before updating it ：

select id,amount,version from user id=123;

Copy

If the data exists , Let's say we find version be equal to 1, Reuse id and version Fields update data as query criteria ：

update user set amount=amount+100,version=version+1
where id=123 and version=1;

Copy

While updating the data version+1, Then judge this time update The number of rows affected by the operation , If it is greater than 0, The update is successful , If it is equal to 0, This update does not change the data .

Because of the first request version be equal to 1 It can be successful , After successful operation version become 2 了 . At this time, if concurrent requests come over , Do the same thing again sql：

 update user set amount=amount+100,version=version+1
where id=123 and version=1;

Copy

The update Operations don't really update data , Final sql The number of rows affected by the execution result of is 0, because version Has become a 2 了 ,where Medium version=1 It's definitely not going to meet the conditions . But to ensure interface idempotency , Interface can return success directly , because version The value has been modified , Then we must have succeeded once before , It's followed by repeated requests .

The specific process is as follows ：

Specific steps ：

1. First, according to id Query user information , contain version Field
2. according to id and version Field value as where Parameters of the condition , Update user information , meanwhile version+1
3. Determine the number of rows affected by the operation , If it affects 1 That's ok , It is a request , You can do other data operations .
4. If it affects 0 That's ok , The description is a repeat request , Then directly return to success .

4. Add unique index

In most cases , To prevent duplicate data , We all add a unique index to the table , This is a very simple , And effective solutions .

alter table `order` add UNIQUE KEY `un_code` (`code`);

Copy

With a unique index , The first request data can be inserted successfully . But the same request later , When inserting data, it will report Duplicate entry '002' for key 'order.un_code abnormal , Indicates that there is a conflict in the unique index .

Although throwing anomalies has no effect on the data , No false data . But to ensure interface idempotency , We need to catch the exception , Then return to success .

If it is java The program needs to capture ：DuplicateKeyException abnormal , If used spring The framework also needs to capture ：MySQLIntegrityConstraintViolationException abnormal .

The specific flow chart is as follows ：

Specific steps ：

1. The user initiates the request through the browser , The server collects data .
2. Insert the data into mysql
3. Judge whether the execution is successful , If it works , Then operate other data （ There may be other business logic ）.
4. If execution fails , Catch unique index conflict exception , Go straight back to success .

5. Build a weight watch

Sometimes not all scenarios in the table are allowed to generate duplicate data , Only certain scenes are not allowed . Now , Add a unique index directly to the table , It's obviously not appropriate .

In this case , We can go through Build a weight watch To solve the problem .

The table can contain only two fields ：id and unique index , The unique index can be multiple fields, such as ：name、code And so on , for example ：susan_0001.

The specific flow chart is as follows ：

Specific steps ：

1. The user initiates the request through the browser , The server collects data .
2. Insert the data into mysql Counterweight gauge
3. Judge whether the execution is successful , If it works , Then do mysql Other data operations （ There may be other business logic ）.
4. If execution fails , Catch unique index conflict exception , Go straight back to success .

Here's the thing to watch out for ： The anti duplication table and the business table must be in the same database , And the operation should be in the same transaction .

6. According to the state machine

Most of the time, business tables are stateful , For example, the order form has ：1- Place an order 、2- Paid 、3- complete 、4- Revocation, etc . If the values of these states are regular , According to the business node is just from small to large , We can use it to ensure the idempotency of the interface .

If id=123 Your order status is Paid , Now it's going to be complete state .

update `order` set status=3 where id=123 and status=2;

Copy

On the first request , The status of the order is Paid , The value is 2, So the update Statement can update data normally ,sql The number of rows affected by the execution result is 1, The order status becomes 3.

The same request came later , Do the same thing again sql when , Because the order status becomes 3, Reuse status=2 As a condition , Unable to find the data that needs to be updated , So the final sql The number of rows affected by the execution result is 0, It doesn't really update the data . But to ensure interface idempotency , The number of rows affected is 0 when , Interface can also return success directly .

The specific flow chart is as follows ：

Specific steps ：

1. The user initiates the request through the browser , The server collects data .
2. according to id And the current state as conditions , Update to the next state
3. Determine the number of rows affected by the operation , If it affects 1 That's ok , Indicates that the current operation is successful , You can do other data operations .
4. If it affects 0 That's ok , The description is a repeat request , Go straight back to success .

The main special attention is , The scheme is limited to the ones to be updated The table has status fields , And it's just about to be updated Status field This special case of , Not all scenarios apply .

7. Add distribution lock

As a matter of fact, what I mentioned before Add unique index perhaps Add weight watch , The essence is to use database Of Distributed lock , It is also a kind of distributed lock . But because of Database distributed lock My performance is not very good , We can use ：redis or zookeeper.

In view of the fact that many companies now use the distributed configuration center apollo or nacos, It's rarely used zookeeper 了 , We use redis As an example, introduce distributed lock .

At present, there are mainly three ways to achieve redis Distributed locks for ：

1. setNx command
2. set command
3. Redission frame

Each has its own advantages and disadvantages , I won't talk about the specific implementation details , Interested friends can chat with me on wechat .

The specific flow chart is as follows ：

Specific steps ：

1. The user initiates the request through the browser , The server will collect data , And generate the order number code As the only business field .
2. Use redis Of set command , Put the order code Set to redis in , At the same time, set the timeout .
3. Judge whether the setting is successful , If the setting is successful , The description is the first request , Then data operation is performed .
4. If the setup fails , The description is a repeat request , Then directly return to success .

Here's the thing to watch out for ： Distributed locks must have a reasonable expiration time , If the setting is too short , Unable to effectively prevent duplicate requests . If the setting is too long , It could be wasted redis Storage space , It depends on the actual business situation .

8. obtain token

In addition to the above options , There's a last use token The plan . This scheme is a little different from all previous schemes , It takes two requests to complete a business operation .

1. First request for token
2. Second request with this token, Complete business operations .

The specific flow chart is as follows ：

First step , First get token.

The second step , Do specific business operations .

Specific steps ：

1. When the user visits the page , The browser automatically initiates the acquisition token request .
2. Server generation token, Save to redis in , And then back to the browser .
3. When a user makes a request through a browser , Take with you token.
4. stay redis Query the token Whether there is , If it doesn't exist , The description is the first request , Do the following data operation .
5. If there is , The description is a repeat request , Then directly return to success .
6. stay redis in token After the expiration time , Automatically deleted .

The above scheme is designed for idempotent .

If it's weight proof design , The flow chart needs to be changed ：

Here's the thing to watch out for ：token It has to be the only one in the world .

Publisher ： Full stack programmer stack length , Reprint please indicate the source ：https://javaforall.cn/111566.html Link to the original text ：https://javaforall.cn