How does yyds dry inventory deal with repeated messages in the consumption process?

If the message delivery fails , Then the sender will retry , Retry may result in duplicate messages . If you don't process duplicate messages , May be a surprise . For example, a consumer order message , Statistics of the amount of micro service orders . If duplicate messages are not handled correctly , There will be repeated Statistics . That only depends on MQ Can you guarantee that the message is not repeated ？
Message repetition must exist , stay MQTT agreement , Give three ways to deliver messages

1 Service quality standards

Service quality from low to high ：

At most once

One more time . When a message is delivered , Be served at most once . That is, there is no message reliability guarantee , It's allowed to lose news . Generally, it is used in some monitoring scenarios that do not require high reliability of messages , For example, report the temperature data of the computer room once a minute , A small amount of data loss is acceptable

At least once

At least once . When a message is delivered , It will be delivered at least once . That is, it is not allowed to lose messages , But a small number of duplicate messages are allowed

Exactly once

Exactly Once . When a message is delivered , It will only be delivered once , No loss or repetition is allowed
Service quality standards are not only suitable for MQTT, For all MQ All applicable . Most of the MQ The quality of service provided is At least once, Such as RocketMQ、RabbitMQ and Kafka. so to speak MQ By itself, there is no guarantee that the message will not repeat .

You are not right , I have ever seen Kafka file ,Kafka Support Exactly once Of ！

you 're right ,Kafka I do support Exactly once, But what this article says is no problem .Kafka Of “Exactly once” And messaging quality of service standards “Exactly once” Different , It is Kafka Another feature provided ,Kafka The transactions supported in are also different from those commonly understood .Kafka Business and Excactly once Mainly for the calculation of combined flow .
since MQ There is no guarantee that messages are not repeated , You have to accept the consumer code “ The message may be duplicate ” This reality , Solve the impact of repeated messages on business through business code .

2 Idempotency

The general solution to duplicate messages is on the consumer side , Make the operation of consuming messages idempotent （Idempotence）：
Describe an operation 、 Method or service , The impact of any multiple execution is the same as that of one execution .
An idempotent method , Use the same parameters , Make multiple calls to it and one call , Have an impact on the system . therefore , For idempotent methods , There is no need to worry that repeated execution will change the system .

Example

Don't think about concurrency ,“ Put the account X The balance of is set to 100 element ”, The impact on the system after one execution is , Account X The balance becomes 100 element . Just provide parameters 100 Yuan doesn't change , How many times , Account X The balance is always 100, This operation is an idempotent operation .
“ Put the account X Balance plus 100 element ”, This operation is not idempotent , Every execution , Increase in account balance 100, The impact of multiple and one executions on the system （ Account balance ） Different .
If the business logic of the system consumption message is idempotent , Then don't worry about repeated messages , Because of the same news , One consumption has the same impact on the system as many times . That is, multiple consumption is equal to one consumption .
From the impact on the system ：At least once

• System influence result ：At least once + Idempotent consumption = Exactly once.

3 Idempotent implementation

It's best to start with business logic , Design the consumption business as an idempotent operation . But not all businesses are naturally idempotent , It takes some skill .

3.1 Database unique constraints

For example, for ： Put the account X Balance plus 100.
It can be restricted to each transfer order , Each account can only be changed once . The simplest , stay DB Zhongjian one 【 Transfer statement 】：

• Transfer form ID
• Account ID
• Change amount

And then to 【 Transfer form ID, Account ID】 Join together to create unique constraints , In this way, the same transfer order ID、 Account ID, There is at most one record in the table .
The consumption message logic can be changed to ：“ stay 【 Transfer statement 】 Add a transfer record , Then according to the transfer record , Update user balance asynchronously .”
In the operation of adding a transfer record to the transfer flow table , because 【 Transfer form ID, Account ID】 Unique constraint , For the same transfer order , Only one record can be inserted into the same account , Subsequent repeated insert operations will fail , This implements idempotence .
therefore , As long as it supports something like “INSERT IF NOT EXIST” Semantic storage systems can achieve idempotence .
such as , You can use

Redis Of SETNX

Replace the unique constraint in the database , Achieve idempotent consumption .

3.2 Set preconditions for updated data （ similar CAS）

Set a precondition for data changes ：

• Update the data when the conditions are met
• Otherwise, refuse to update the data

When updating data , At the same time, change the data to be judged in the precondition . therefore , When repeating this operation , Because when the data is updated for the first time , The judgment data in the preconditions has been changed , Does not meet the preconditions , No more updates will be performed .
“ Put the account X The balance of increased 100 element ”, This operation adds a precondition , Turn into ：“ If account X The current balance is 500 element , Add the balance to 100 element ” Is idempotent . Corresponding to MQ news , Carry the current balance in the message body , Consumption time judgment DB Current balance in == The balance in the message , Update only when equal .
But the data to be updated is not numeric , Or do a complex update operation ？ What are the pre judgment conditions ？

MVCC

More general , Is to add a version number to the data version attribute , Before each data update , Compare
Current data version == In the news version

• atypism , Reject updates
• Agreement , Update the data and the version number +1, Idempotent update can be realized

3.3 Record and check operation

If neither of the first two options applies , And the most versatile 、 The most widely applicable scheme ： Record and check operation , Also known as “Token Mechanism or GUID（ Globally unique ID） Mechanism ”, Before performing the data update operation , First check whether the update operation has been performed .

• When sending messages , Assign global uniqueness to each message ID
• Consumption time , First, according to ID Check whether the message has been consumed , If there is no , Update the data and set the consumption status to consumed

But it is difficult to realize in distributed system ：

• First , Assign a globally unique to each message ID, There are many ways , But it's not so good, and it's simple 、 High availability and performance , There are more or less sacrifices
• More trouble ,“ Check consumption status , Then update the data and set the consumption status ”, The three operations must be a group of operations , Guaranteed atomicity , In order to realize idempotency , Otherwise, it would be Bug

For example, for the same message ：“ overall situation ID by 8, Operation for ： to ID by 666 Account increase 100 element ”, This may happen ：

• t0 moment ：Consumer A Got a message , Check message execution status , Found that the message has not been processed , Start execution “ Account increase 100 element ”
• t1 moment ：Consumer B Got a message , Check message execution status , Found that the message has not been processed , Because this moment ,Consumer A We haven't had time to update the message execution status
• This leads to the account being wrongly increased twice 100 element , This is a very easy mistake to make in a distributed system

Regarding this , It can be realized by transaction , You can also lock , But in Distributed Systems , Distributed transactions 、 Distributed locks introduce high complexity . So it's not recommended .

summary

These idempotent schemes can not only be used to solve the problem of duplicate messages , It can also solve the problem of repeated requests or repeated calls . such as ：

• take HTTP Services are designed to be idempotent , Solve the front-end or APP The problem of submitting form data repeatedly
• Design a microservice as idempotent , solve RPC Repeated calls caused by automatic retrying of framework

why MQ They only provide At least once Service quality , Instead of Exactly once

if MQ Realization exactly once, May trigger ：

• The consumer end pull when , Check whether this message is consumed , This detection mechanism will undoubtedly slow down message consumption . As the news soared , Consumption performance is bound to decline sharply , Leads to a backlog of messages
• The inspection mechanism needs the cooperation of the business side , If a message does not return for a long time ack,MQ We need to go back and look at the consumption results （ Similar to the back check mechanism of transaction messages ）. This increases the pressure and unknown factors on the business side .
• To ensure that messages are not lost or duplicated , The queue needs to take some means similar to back checking , Check whether the consumer has received the message for processing , To some extent, it will lead to a series of problems such as queue accumulation , And the complexity of queue implementation increases
• From the consumer's point of view , Because consumers and Broker Service Each end will have its own cluster , There may be network jitter on the consumer side , Lead to Broker Service To ensure that messages are not lost or duplicated , You need to always perform similar operations of back checking , But because of network problems , Cause the queue to pile up .

therefore ,MQ Don't realize exactly once, It is at least once

• MQ Don't realize exactly once, It is at least once + Idempotency , The idempotency is handled by the business code on the consumer side .

MQ Even if you do Exactly once Level ,Con Also do idempotent . because Con from MQ When getting the message , if Con Consumption success , but ack Failure ,Con Still get duplicate messages , therefore MQ Work hard to make Exactly once The problem of duplicate messages on the business side cannot be avoided .

Use DB The unique index of prevents messages from being consumed repeatedly , If there are sub databases and sub tables in the business system , Consumption messages are routed to different libraries or tables , There will still be problems ？

Generally, there will be no problem , Because using our method , A specific message , It will always fall to the determined library table , The repeated messages will also be displayed in the same database table .

If the queue implements At least once, But in order not to lose news ,Broker Service Will try again , But it's impossible to keep trying again , How to handle if you keep trying again or fail ？

yes , we have MQ There will be a special queue , Saving these is always a failure of consumption “ The bad news ”, Then continue the news after consumption , Avoid these bad news getting stuck in the queue . This kind of bad news is generally not caused by network reasons or consumer downtime , Mostly because the message data itself has problems , The consumer's business logic cannot handle .
exactly once, Achieve performance loss , When concurrency is high, message accumulation is easy to occur ; The original intention of message queue design is to solve the problem of decoupling , Decoupled objects tend to be highly concurrent , High performance requirements , From the perspective of product demand , Message queue design pays more attention to performance , Not precision （exactly once）; From an infrastructure perspective , The focus is on the demand that accounts for a large proportion （ I can't help sending , You can resend it ）, Demand with a small proportion （ Sensitive , Trigger only once ） It can be extracted separately to realize . Last , Ask the teacher if there is any specific business scenario , Not with this exactly once Impossible