MQ message queue is used to design the high concurrency of the order placing process, the generation scenarios and solutions of message squeeze, message loss and message repetition

Use message queue to design the order process with high concurrency

Use message queue to design the order process with high concurrency

Using message queues requires solving a core problem , That is how to ensure the reliability of the message , There are three solutions , In microservice architecture projects, businesses with high concurrency cannot be used seata To handle distributed transactions , So use message queuing to ensure the final consistency of data . Flexible transactions using message transactions ： Reliable information + The final consistency plan （ Asynchronous assured ）

Lost message

Send to mq The message of will be lost in those scenarios ：

1、 Because of network fluctuations , The network flash outage caused the message not to arrive at the server , Didn't send out , It can be done at the place where the message is sent try catch, Make sure that the message will be sent , Log messages that fail to be sent , Whether the status of each message is received by the server should be recorded , And this message should be stored in a persistent database, such as mysql Database etc. . And regularly resend , If the message is not sent successfully , Scan the database regularly and resend unsuccessful messages .

Create one for each database mq_message Message table , Used to store sent messages , Save the details of each message to the database , After saving, as long as the sending fails , You can scan the database regularly and send the failed message again .

​
CREATE TABLE `mq_message` (
  `message_id` char(32) NOT NULL,        # The only news id
  `content` text,      # The content of the message   convert to json
  `to_exchane` varchar(255) DEFAULT NULL,  # The switch sent to 
  `routing_key` varchar(255) DEFAULT NULL,   # What is the routing key sent 
  `class_type` varchar(255) DEFAULT NULL,    # Mark the type of message 
  `message_status` int(1) DEFAULT '0' COMMENT '0- newly build  1- Has been sent  2- Wrong arrival  3- Arrived ',
  `create_time` datetime DEFAULT NULL,
  `update_time` datetime DEFAULT NULL,
  PRIMARY KEY (`message_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

​2、 The message arrived Broker,Broker To write a message to disk （ Persistence ） Success is considered. , here Broker Persistence has not been completed , Downtime .mq After the server receives the message, it must be persisted , Then send the message to the switch , The switch is handed over to the designated queue .
​
In addition to consumer information, there is a confirmation mechanism , There is also an acknowledgement mechanism for sending messages , Reliable arrival -ConfirmCallback

Message arrival Queue after , Only then can the message arrive successfully ,publisher The confirmation callback mechanism must also be added , Confirm the successful message , Then modify the status of the message corresponding to the database .

3、 Automatically ACK Under the state of , Consumers receive messages , But there was no time to consume the news , The consumer server goes down , Cause message loss , So be sure to turn on manual ACK, Only after successful consumption , Fail or fail before you have time to deal with it noAck Rejoin the team .

The above three points can actually belong to one point , That is to do a good job in the message confirmation mechanism （ Both the sender and the consumer should confirm the arrival and consumption of the message ）, Every message sent is recorded in the database , Send the failed message again regularly .
​​​​
Message arrival Queue after , Only then can the message arrive successfully ,publisher The confirmation callback mechanism must also be added , Confirm the successful message , Then modify the status of the message corresponding to the database .

3、 Automatically ACK Under the state of , Consumers receive messages , But there was no time to consume the news , The consumer server goes down , Cause message loss , So be sure to turn on manual ACK, Only after successful consumption , Fail or fail before you have time to deal with it noAck Rejoin the team .

The above three points can actually belong to one point , That is to do a good job in the message confirmation mechanism （ Both the sender and the consumer should confirm the arrival and consumption of the message ）, Every message sent is recorded in the database , Send the failed message again regularly .

Duplicate message

Message repetition is a message sent to consumers twice , It is equivalent to that consumers have consumed twice

scene ：

1、 After the consumer executes the core code , The transaction has been committed ,ack when , Server down , Result in not giving mq Respond to , No, ack success , Because it is opened manually ACK, So the message queue will think it failed ,Broker The message was re sent by unack Turn into ready, And send it to other consumers , Send the message again , But the consumer has finished executing the core code , This leads to repeated consumption

2、 Message consumption failed , Due to the retry mechanism , Automatically send the message again , This is a normal scene

Message successful consumption , however ack Time goes down , Message by unack Turn into ready,Broker Re send to consumers , This leads to repeated message consumption , The solution is to design idempotent with the core business , Use the uniqueness of each message ID To record whether the core business has been executed （ After committing the transaction ）, Make the message unique ID The status flag of is consumed , Consume again according to the unique ID To determine whether it can be consumed .

There's a backlog of news

Message backlog is , There are too many messages in the message queue , It will definitely affect mq Performance of , bring mq Performance degradation of

scene ：

1、 Consumer downtime ,mq Not connected to any consumer , Or even too little , There are no consumers to consume news , Producers are also constantly generating messages

2、 The traffic sent is too large ,mq There are too many messages saved

solve ：

1、 More consumers online , For normal consumption

2、 If the business volume is too large , Too much data , Then you can launch a consumer dedicated to processing messages , This consumer will mq Take out all the information inside , And then put it in the database , Write an offline processing business by yourself , Process messages in the database one by one

summary

Message loss is the most important thing to avoid , This one needs closed loop

Finally, the last message queue high concurrency design flowchart for the order placing process , as follows