MQ prevent message loss and repeated consumption

RabbitMQ How to prevent message loss and repeated consumption

One 、 Lost message

As shown in the figure ,RabbitMQ The lost message can be sent to any node . Remember the execution flow of the following figure

1.1、 The producer failed to send the message to MQ

The reason for the loss ： Because of the instability of network transmission , When the producer is talking to MQ In the process of sending a message ,MQ Failed to receive message , But producers think MQ Successfully received message , The message will not be sent again , This leads to the loss of messages .

terms of settlement ： There are two solutions ： Affairs and mechanisms confirm Mechanism , The most common is confirm Mechanism （ Release confirmation mechanism ）

Be careful ：

1、RabbitMQ The transaction mechanism is synchronous , Very energy consuming , It will reduce RabbitMQ Throughput .
2、confirm The mechanism is asynchronous , After the generator sends a message , No need to wait RabbitMQ The callback , You can send the next message , When RabbitMQ After successfully receiving the message, it will automatically and asynchronously call back to an interface of the producer to return the success message .

The two mechanisms are described below ：

confirm（ Release confirmation ） Mechanism : RabbitMQ Can be opened confirm Pattern , Set on at the producer confirm After the pattern , Every time a producer writes a message, it assigns a unique id, If the message is successfully written RabbitMQ in ,RabbitMQ Will send back a message to the producer ack news , Tell you the news ok 了 . If RabbitMQ It didn't work , Will call back one of you nack Interface , Tell you the message failed to receive , Producers can once again report to RabbitMQ Send a message .

RabbitMQ Provides transaction functions , Open before producer sends data RabbitMQ Business channel.txSelect, And then send a message , If the message doesn't succeed RabbitMQ Received , Then the producer will receive an exception , At this point, you can roll back the transaction channel.txRollback, Then try sending the message again ; If you receive a message , Then you can commit the transaction channel.txCommit. The pseudocode is as follows ：

//1、 Open transaction 
channel.txSelect 
try {
    
//2、 Send a message here  
} catch (Exception e) {
    
	channel.txRollback
 	//3、 Here is the message again 
}
//4、 Commit transaction 
channel.txCommit

1.2、RabbitMQ After receiving the message, the message is lost

The reason for the loss ：RabbitMQ Received a message from the producer , It's in memory , If it is not sent to consumers , here RabbitMQ It's down. , So when you start again , The messages in the original memory are lost .

terms of settlement

1、 Turn on RabbitMQ The persistence of . When the producer successfully writes the message to RabbitMQ after ,RabbitMQ Just persist the message to disk . Combined with the above confirm Mechanism , If message persistence is enabled , Only after the message is successfully persisted to the disk , Will call back the producer's interface and return ack news , Otherwise, it's a failure
2、 After reboot , He will read messages from the disk , It will not lead to the loss of messages .

Persistent configuration ：

1、queue Queue persistence , establish queue Set it to persistent , Even if you restart at this time rabbitmq queue , It still exists , But it won't last queue Of the data in .

2、Message Message persistence , When sending a message deliveryMode Set to 2, Set to persistent , here RabbitMQ It will persist the message to disk .
3、 Be careful ： For persistence to work, you must set both persistence settings at the same time

1.3、 The consumer lost the message

The reason for the loss ： If RabbitMQ Successfully sent the message to the consumer , that RabbitMQ Of ack The mechanism will automatically return success , Indicates that the message was sent successfully , This message will not be sent next time . But if at this time , The consumer hasn't finished processing the message yet , Then it went down , Then the message is lost .

The solution ： Simply speaking , It must be closed RabbitMQ Automatic ack, Use manual ack（ Message answering mechanism ）. In this case , If you haven't dealt with it , No, No. ack 了 ！ that RabbitMQ Think you're not done with it , When they arrive in Timeout time Not yet submitted , This is the time RabbitMQ Think you failed to deal with , Messages automatically rejoin the queue , Will allocate this consumption to others consumer To deal with . When the processing is successful rabbitmq Will delete the message

Message answering mechanism , Divided into two ： Auto answer 、 Manual response .

Two 、 How to prevent repeated consumption

Under normal circumstances , When consumers are consuming news , After consumption , A confirmation message will be sent to RabbitMQ（ If RabbitMQ I didn't receive it ）, The message queue knows that the message has been consumed , The message is removed from the message queue .
Under special circumstances ：

Because of network transmission and other failures , If this confirmation message is not sent back to RabbitMQ,RabbitMQ It may be thought that the consumer failed to handle , The message will be distributed to other consumers again .

The solution is ： Keep the message unique , Even if it's multiple transmissions , Don't let the multiple consumption of news have an impact ; Ensure the idempotency of the message ;

1、 At the time of message production ,MQ Internally, a message is generated for each producer inner-msg-id, As a basis for de duplication and idempotency , Avoid duplicate messages entering the queue

2、 When the message is consumed , It is required that there must be a bizId（ Globally unique : If paid ID、 Order ID etc. ） As a basis for de duplication and idempotency , Avoid repeated consumption of the same message .

So under the condition of ensuring stability ： Better up 2 Do all of them , Then when you spend , Also put msgId Put in storage , In consumption, it is to query whether it has been consumed , If there is one, don't deal with it

Introduction to this article ：
https://blog.csdn.net/wang_luwei/article/details/123613091