Distributed transactions: the final consistency scheme of reliable messages

We must be familiar with business , For example, it is often mentioned ACID, But for the content of subsequent distributed transactions , Let's talk first ACID, Then I will introduce what is distributed transaction , Finally, we will focus on the distributed transaction solution based on reliable messages .

What is business

Strictly speaking, transactions should be atomic 、 Uniformity 、 Isolation and persistence , abbreviation ACID.

1. Atomicity （Atomicity）, It can be understood that all operations within a transaction are executed , Either not .
2. Uniformity （Consistency）, It can be understood that the data satisfies the integrity constraints , That is, there will be no intermediate state data , For example, your wallet has 100, I have... In my wallet 100, You call me 50 block , The money in your wallet should be 50, The money in my wallet should be 150, It won't exist. I'll add my money , Your middle state without deduction .
3. Isolation, （Isolation）, It means that multiple transactions will not interfere with each other when they are executed simultaneously , In other words, the data in one transaction is isolated from other transactions .
4. persistence （Durability）, It means that after a transaction is completed, the data is saved forever , Any subsequent operations or failures will not affect the outcome of the transaction .

In the popular sense, transaction is to make some update operations successful , Or they all failed .

What is distributed transaction

As the name implies, distributed transaction is to implement transaction in distributed system , It is actually a combination of multiple local transactions .

A large operation consists of different small operations , These small operations are distributed on different servers , Distributed transactions need to ensure that these small operations are all successful , All or nothing . essentially , Distributed transaction is to ensure the data consistency of different databases .

There are several common solutions for distributed transactions :2PC,3PC,TCC, Local message table 、 The final consistency of reliable information 、 Try your best to inform, etc

Today we will focus on the solution to the final consistency of reliable messages

What is the ultimate consistency scheme for reliable messages

The final consistency scheme of reliable messages refers to sending messages to the message middleware after the transaction initiator completes the local transaction , Transaction participants ( Message consumer ) You must be able to receive the message and process the transaction successfully , This scheme emphasizes that as long as the message is sent to the transaction participant , Then the final transaction must be consistent .

What are the problems with this approach ？

This scheme is implemented through message oriented middleware , Transaction initiator （ Message producer ） Send messages to message middleware , Transaction participants receive messages from message middleware , Due to the uncertainty of network communication, it will lead to distributed transaction problems , Here's the picture ：

1. Local transactions and atomicity of messages

As shown in the figure above, it is in the dotted box , There are several situations ：

1） Local transaction commit failed , The message is not sent .
2） Local transaction succeeded , Message delivery failed , Local transaction rollback .
3） Local message succeeded , Message timeout , Local transaction rollback , The message eventually failed .
4） Local message succeeded , Message timeout , Local transaction rollback , The message finally succeeded .

in summary , There is a fourth case , Cause local transactions , Inconsistent with the transaction of the message party .

1. Reliability of message received by transaction participants .
   

Message oriented middleware and transaction participants should ensure that messages can be successfully consumed .

1. Message re consumption
   

Note the idempotency of the interface of the transaction participants , Message participants may have successfully consumed , Due to network problems, message middleware thinks that messages are not consumed , Problems after initiating retry .

Solution

1. Local message table
   

The key to the local message table is that there is a local record table for storing message logs , You need to start a scheduled task to constantly scan message logs , Ensure that the message can be sent . The specific process is shown in the following figure ：

The flow chart above ：

1） The local transaction of the transaction initiator was executed successfully , Record the message log in the local message table .
2） Start timing task , Loop scan local message table .
3） When the scheduled task scans the message, it sends the message to the message middleware .
4） The message middleware receives a message , The success return message sends a success notification to the transaction initiator .
5） If the transaction initiator receives the message and sends it successfully, the log message will be deleted .
6） Transaction participants subscribe to messages , News consumption .
7） Transaction participants handle local transactions .
8） Local transaction successfully , Send successfully ack To message middleware .

Something to watch out for :
Transaction participants guarantee the idempotency of the interface .

1. RocketMq Transaction message scheme
   

Apache RocketMQ 4.3 Later versions officially support transaction messages , It provides convenient support for distributed transaction implementation . stay RocketMQ 4.3 After the implementation of the complete transaction message , In fact, it is an encapsulation of the local message table , Moved the local message table to MQ Inside , solve Producer The atomicity of message sending and local transaction execution at the end .

Implementation process ：

1） The transaction initiator sends Half Transaction message
2）RocketMq reply Half Send successfully
3） The transaction initiator executes a local transaction
4） The transaction initiator successfully executed the local transaction , send out commit To RocketMq,mq Post messages to transaction participants ; Transaction initiator failed to execute local transaction , send out rollback To RocketMq,mq removal message .
5） When RocketMq No confirmation from the transaction initiator has been received within a certain period of time , The transaction initiator will be checked back .
6） The transaction initiator queries the local transaction status .
7） The transaction initiator sends a message according to the queried transaction status commint/rollback To RocketMq.
8） When RocketMq launch commit after , Failed to receive or failed to receive within a certain period of time ack, A retry will be initiated .

advantage ：

Message data is stored independently , Reduce the coupling between business system and message system .
Throughput is better than the local message table scheme .

shortcoming ：

Two network requests are needed to send a message (half news + commit/rollback).
You need to implement the message retrieval interface .

In fact, every distributed transaction solution has its advantages and disadvantages , We need to weigh the pros and cons , Choosing the most appropriate business scenario is the king ！

Okay . That's all for today , I will continue to share what I have learned and thought , I hope we are on the road to success together ！