Essential for back-end programmers: distributed transaction Basics

Want to make APP Same thing as WeChat , Can run small programs smoothly ？ | Experience will send you to Xinjiang 、 Huawei 、 Cherry keyboard ！>>>

Preface

Recently I read several blogs about distributed transactions , Take notes . ha-ha ~

Database transactions

Database transactions （ abbreviation ： Business ）, It is a logical unit in the execution of database management system , It consists of a limited sequence of database operations , These operations are either all performed , Or none at all , It is an indivisible work unit .

Several typical features of database transactions ： Atomicity (Atomicity )、 Uniformity ( Consistency )、 Isolation, ( Isolation) And persistence (Durabilily), The abbreviation is ACID.

• Atomicity ： The transaction is executed as a whole , The operations on the database contained in it are either all performed , Either not .
• Uniformity ： Before and after the transaction , The data will not be destroyed , If A Account to B Account transfer 10 Yuan , Whether it works or not ,A and B The total amount of is the same .
• Isolation, ： When multiple transactions are accessed concurrently , Transactions are isolated from each other , That is, one transaction does not affect the running effect of other transactions . in short , It's just that the water doesn't flow into the river .
• persistence ： Indicates that after the transaction is completed , Operational changes made by the transaction to the database , Will persist in the database .

How transactions are implemented

Local transactions

Traditional single server , Transactions in a single relational database , It's local business . Local transactions are managed by resource manager ,JDBC Transaction is a very typical local transaction .

Transaction log

innodb The transaction log includes redo log and undo log.

redo log（ Redo log ）

redo log Usually physical logs , What is recorded is the physical modification of the data page , Not how to modify a line or lines , It is used to recover the physical data page after submission .

undo log（ Rollback log ）

undo log It's a logical log , and redo log It's different to record physical logs . Think of it this way , When delete When a record ,undo log A corresponding insert Record , When update When a record , It records an opposite update Record .

Business ACID The realization idea of features

• Atomicity ： It's using undo log To achieve , If there is an error during the transaction execution or the user executes rollback, The system goes through undo log The log returns the status of the start of the transaction .
• persistence ： Use redo log To achieve , as long as redo log Log persistence , When the system crashes , You can pass redo log Recover data .
• Isolation, ： Through locks and MVCC, To separate affairs from each other .
• Uniformity ： By rolling back 、 recovery , And isolation in the case of concurrency , To achieve consistency .

  Distributed transactions

Distributed transactions ： It refers to the participants in the transaction 、 Servers that support transactions 、 Resource servers and transaction managers are located on different nodes of different distributed systems . Simply speaking , Distributed transaction refers to the transaction in distributed system , It exists to ensure the data consistency of different database nodes .

Why do I need distributed transactions ？ Next, I will explain it in two aspects ：

Distributed transaction under microservice architecture

With the rapid development of Internet , Lightweight and well-defined microservices , On the stage of history . such as , A user places an order , Buy live gift service , Be divided into three service, They are gold coin service （coinService）, Order service （orderService）、 Gift service （giftService）. These services are deployed on different machines （ node ）, Corresponding database （ Gold coin database 、 Order database 、 Gift database ） Also on different nodes .

Users place an order to buy gifts , Gift database 、 Gold coin database 、 The order database is on different nodes , It's not possible to use local transactions , So how to guarantee different databases （ node ） Data consistency on ？ This requires distributed transactions ~

Distributed transactions under sub databases and sub tables

As the business grows , The data in the database is getting bigger and bigger , More than ten million levels of data , We need to divide it into databases and tables （ The company used to use mycat Sub database and sub table , Later used sharding-jdbc）. A bank , The data is distributed on different nodes , For example, some of them are in Shenzhen computer room , Some are in Beijing computer room ~ You want to use local affairs to guarantee , I've been indifferent ~ You still need distributed transactions .

such as A turn 10 Block to B,A The account data is in Beijing computer room ,B The account data of is in Shenzhen computer room . The process is as follows ：

CAP theory &BASE theory

Learn about distributed transactions , Of course, we need to understand CAP Theory and BASE theory .

CAP theory

CAP Theory as the basic theory of distributed system , In a distributed system , Consistency（ Uniformity ）、 Availability（ Usability ）、Partition tolerance（ Partition tolerance ）, These three elements can only achieve two points at most .

Uniformity (C：Consistency)：

Consistency is the ability of data to be consistent across multiple copies . For example, a data is updated after a partition node , The data read from other partition nodes is also the updated data .

Usability (A：Availability)：

Availability means that the services provided by the system must always be available , For each operation request of the user, the result can always be returned in a limited time . Here's the point " In a limited time " and " Return results ".

Partition tolerance （P:Partition tolerance）:

Distributed systems encounter any network partition failure , There is still a need to be able to ensure that external services meet the requirements of consistency and availability .

choice explain
CA Give up partition fault tolerance , Enhance consistency and availability , In fact, it is the choice of traditional stand-alone database
AP Give up consistency , Partition fault tolerance and availability , This is the choice of many distributed systems when they are designed
CP Discard availability , Pursue consistency and partition fault tolerance , Network problems will directly make the whole system unavailable

BASE theory

BASE theory , It's right CAP in AP An extension of , For our business system , We consider sacrificing consistency for system availability and partition fault tolerance .BASE yes Basically Available( Basic available ),Soft state（ Soft state ）, and Eventually consistent（ Final consistency ） Abbreviations of three phrases .

Basically Available

Basic available ： It is realized by supporting local fault rather than global fault of the system . If you partition users in 5 Database server , The failure of a user database only affects the specific host 20% Users of , Other users are not affected .

Soft State

Soft state , The state can be out of sync for a while

Eventually Consistent

Final agreement , Finally, the data is consistent , Instead of being consistent all the time .

Several solutions for distributed transactions

Distributed transaction solutions mainly include the following ：

• 2PC( Two stage submission ) programme
• TCC（Try、Confirm、Cancel）
• Local message table
• Best effort notification
• Saga Business

  Two stage proposal

Two phase commit is a common distributed transaction solution . The commit of a transaction is divided into two phases ： Preparation phase and submission of implementation plan .

Successful submission of phase II

Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success .

Submission execution phase , If the transaction manager receives a success message from all resource managers , Then send a submit message... To each resource manager ,RM according to TM The order execution submission of . Pictured ：

Failure of two-stage submission

Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success , If execution fails , Then it returns failure .

Submission execution phase , If the transaction manager receives a message that any resource manager fails , Then send a rollback message to each resource manager . The resource manager rolls back the local transaction operation according to the instructions of the transaction manager , Release all lock resources used in transaction processing .

Advantages and disadvantages of two stage submission

2PC It's easy to implement , Lower cost , But it has the following disadvantages ：

• A single point of the problem ： If the transaction manager fails , Explorer will always be locked .
• Performance issues ： All resource managers are in a synchronous blocking state during the transaction commit phase , Occupy system resources , Until the submission is complete , To release resources , It is easy to cause performance bottleneck .
• Data consistency issues ： If some resource managers receive a submitted message , Some didn't receive , It will lead to data inconsistency .

TCC（ Compensation mechanism ）

TCC Compensation mechanism has been adopted , The central idea is this ： For each operation , To register a corresponding confirmation and compensation （ revoke ） operation .

TCC（Try-Confirm-Cancel） Model

TCC（Try-Confirm-Cancel） Distributed transaction is realized by decomposing business logic . For a specific business service ,TCC The distributed transaction model requires the business system to implement three sections of logic ：

try Stage ： Try to execute , Complete all business consistency checks , Reserve necessary business resources .

Confirm Stage ： At this stage, confirm and submit the business , Don't do any tests , because try The stage has been checked , Default Confirm There is no mistake in the stage .

Cancel Stage ： If business execution fails , Then enter the stage , It will release try All business resources occupied by the phase , And roll back Confirm All operations performed in phase .

TCC The distributed transaction model consists of three parts ： Main business services 、 From business services 、 Business activity manager .

• Main business services ： The main business service is responsible for initiating and completing the entire business activity .
• From business services ： From business services is the whole business activity of the participants , Realization Try、Confirm、Cancel operation , For the main business service to call .
• Business activity manager ： The business activity manager manages and controls the entire business activity , Including recording transaction status , Call... From business service Confirm operation , Call... From business service Cancel Operation etc. .
  Let's take the user's purchase of gifts as an example to simulate TCC The process of implementing distributed transactions ：
  

Suppose the user A The balance is 100 Gold coin , The present you have is 5 Duo .A It took 10 Gold coin , Place the order , Buy 10 Rose . balance 、 Order 、 Gifts are in different databases .

TCC Of Try Stage ：

• Generate an order record , The order status is to be confirmed .
• Will the user A The balance in the gold coins of the account is updated to 90, Freeze the gold coin to 10（ Reserve business resources ）
• The number of users' gifts is 5, The quantity to be added in advance is 10.
• Try After success , Then enter Confirm Stage
• Try There's anything wrong with the process , All enter Cancel Stage
  
  

TCC Of Confirm Stage ：

• Order status updated to paid
• Update user balance to 90, It can be frozen as 0
• The number of user gifts is updated to 15, Increase to 0
• Confirm There's anything wrong with the process , All enter Cancel Stage
• Confirm Process execution successful , Then the transaction ends
  
  

TCC Of Cancel Stage ：

• Modify order status to cancelled
• Update user balance back 100
• Update the number of user gifts to 5
  
  

TCC Advantages and disadvantages

TCC The scheme allows the application to customize the granularity of database operation , Reduced lock conflict , Can improve performance , But it also has the following disadvantages ：

• The application is invasive ,try、confirm、cancel All three phases require business logic implementation .
• Need to be based on the network 、 Different rollback strategies are implemented for different failure reasons such as system failure , It's hard to achieve , Generally with the help of TCC Open source framework ,ByteTCC,TCC-transaction,Himly.

  Local message table

ebay At first, the scheme of local message table was proposed , To solve the problem of distributed transactions . At present, this kind of scheme is widely used in the industry , Its core idea is to split distributed transactions into local transactions for processing . Take a look at the basic implementation flow chart ：

Basic realization ideas

The sender ：

• There needs to be a message table , Record information about message status .
• Business data and message tables are in the same database , That is to make sure that they are in the same local business .
• After processing the business data and writing the message table in the local transaction , By writing a message to MQ Message queue .

• The message will be sent to the message consumer , If sending fails , Try again .

  Message consumer ：

• Processing messages in the message queue , Complete your business logic .
• At this time, if the local transaction is successful , It shows that the processing has been successful .
• If the local transaction fails , Then the execution will be retried .
• If it's a business failure , Send a business compensation message to the message producer , Notify to roll back and other operations .

The producer and consumer scan the local message table regularly , Send the unfinished or failed message again . If there is reliable automatic reconciliation and account supplement logic , This scheme is very practical .

advantage & shortcoming ：

The advantage of this scheme is that it solves the problem of distributed transaction well , Achieve final consistency . The disadvantage is that message tables are coupled to the business system .

Best effort notification

What is the biggest notification

Best effort notification is also a distributed transaction solution . Here is an example of enterprise online bank transfer

• The enterprise online banking system calls the front interface , Go to transfer page
• Enterprise online banking calls transfer system interface
• The transfer system completes the transfer processing , Send transfer result notice to enterprise online banking system , If the notification fails , Then the transfer system repeats the notification according to the strategy .
• The enterprise online banking system has not received the notice , Actively call the interface of the transfer system to query the transfer result .
• The transfer system will encounter situations such as refund of foreign exchange , I will come back to check the account regularly .
  Try your best to inform the objectives of the program , That is to say, the party initiating the notification through a certain mechanism , Try your best to inform the receiver of the result of business processing . The best effort notification implementation mechanism is as follows ：
  
  
  

Try your best to inform the solution

Try your best to inform , May adopt MQ Of ack Mechanism .

programme

• 1. The sponsor will send the notice to MQ.
• 2. The receiving party listens MQ news .
• 3. When the receiving party receives the message , Deal with the business , Respond ack.
• 4. If the receiving party does not respond ack, be MQ Meeting interval 1min、5min、10min Wait for a repeat notice .

• 5. The receiving party can use the message proofreading interface , Keep the message consistent .
  Transfer business flow chart ：
  
  The interaction process is as follows ：
  
  
  

• 1、 The user requests the transfer system to transfer .
• 2、 The transfer system completes the transfer , Send the transfer result to MQ.
• 3、 Enterprise online banking system monitoring MQ, Receive transfer result notice , If the message is not received ,MQ Notifications will be sent repeatedly . Transfer result received , Update transfer status .
• 4、 The enterprise online banking system can also actively query the transfer result query interface of the transfer system , Update transfer status .

  Saga Business

Saga Things are done by Hector Garcia-Molina and Kenneth Salem Put forward , Its core idea is to split long transactions into multiple local short transactions , from Saga Transaction coordinator coordination , If it's finished normally, then it's finished normally , If a step fails , The compensation operation is called once in reverse order .

saga brief introduction

• Saga = Long Live Transaction (LLT, Long life affairs )
• LLT = T1 + T2 + T3 + ... + Ti（Ti For local short transactions ）

• Every local transaction Ti There is a corresponding compensation Ci

  Saga Execution order of

• Normal condition ：T1 T2 T3 ... Tn

• Abnormal situation ：T1 T2 T3 C3 C2 C1

  Saga Two recovery strategies

• Recover backward , If any local sub transaction fails , Compensation for completed transactions . Such as the execution sequence of abnormal conditions T1 T2 Ti Ci C2 C1.
• Recover forward , That is, retry the failed transaction , Let's say that at the end of the day, every sub transaction will succeed . Execution order ：T1, T2, ..., Tj( Failure ), Tj( retry ),..., Tn.

for instance , Suppose the user places an order , flowers 10 $to buy 10 More roses , Then there are

T1= Place the order ,T2= Deduct users 10 Yuan ,T3= User plus 10 Rose , T4= Inventory decrease 10 Rose

C1= Cancellation of order ,C2= Add... To the user 10 Yuan ,C3 = Less users 10 Rose , C4= Inventory plus 10 Rose

Suppose the transaction is executed to T4 An unexpected rollback occurred , stay C4 When we need to add the roses back to the stock , I found that the user's rose has been used , This is a Saga A disadvantage of , Problems caused by no isolation between transactions .

You can solve this problem by ：

• In response to ⽤ layer ⾯ Add ⼊ Logic of logic lock .
• Session layer ⾯ Isolation to ensure that the string ⾏ operation .
• The business layer ⾯ Mining ⽤ Pre freezing of funds ⾦ Of ⽅ To isolate this part of the resources ⾦.
• In the process of business operation, by reading the current status in time ⽅ Get updates .

Reference and thanks

• dried food | An article takes you to learn about distributed transactions
• Then someone asks you about distributed transactions , Throw this to him
• Talk about distributed transactions , And the solution
• Mysql Transaction implementation principle
• Detailed analysis MySQL Transaction log (redo log and undo log)
• 《Saga Distributed transaction resolution ⽅ Case and practice 》
• Best effort notification for distributed transaction solutions

Official account number

• If you think it's well written, please give me a compliment + Pay attention to , thank you ~
• At the same time, I am very much expecting my buddies to pay attention to my official account , Later, better dry goods will be introduced slowly ~ Hee hee