Message queue learning -- Concepts

come from ： Message queue core knowledge point e-book . The author's personal official account 「yes Training strategy 」

1、 What is message queuing

In the field of Computer Science , Message queues and mailboxes are software engineering components , It is usually used for thread communication between processes or within the same process . They deliver messages through queues - Transmit control information or content , Group communication system provides similar functions .

Message queue ( Message middleware ) It is a component that uses queues to communicate .

2、 What message queuing does

In essence, it is because of the rapid development of the Internet , Business is expanding , Enabling technology architecture to evolve .

It's often used to implement ： Asynchronous processing 、 Service decoupling 、 flow control .

2.1、 Asynchronous processing

The longer the call link , The slower the response . Compared with inventory deduction and order generation , Point service and SMS service are unnecessary “ In time ”. Therefore, only after the order generation process , Throw a message into the message queue and you can directly return the response . Moreover, the point service and SMS service can consume this message in parallel .

Message queuing can Reduce waiting for requests , It also allows services to be processed asynchronously and concurrently , Improve the overall performance of the system .

2.2、 Service decoupling

The downstream system of orders is constantly expanding , In order to cater to these downstream systems, order services need to be modified frequently , Any change of downstream system interface may affect the order service . Therefore, message queuing is generally used to solve the problem of coupling between systems , The order service inserts order related messages into the message queue , The downstream system subscribes to this topic whoever wants it . In this way, order service is liberated ！

2.3、 flow control

The request of the gateway is put into the message queue first , The back-end service does its best to consume requests in the message queue . Timeout requests can return errors directly .

Some background tasks , No need to respond in time , And the business processing is complex and the process is long , Then the incoming requests are put into the message queue first , Back end services are handled at their own pace .

The above two situations respectively correspond to the situation that producers produce too fast and consumers consume too slowly , Message queues can play a good buffer effect .

2.4、 Be careful

The stability of a middleware system will be reduced by one more layer , The difficulty of operation and maintenance is raised to a higher level . So weigh the pros and cons , The system is evolving .

3、 Message queuing model

There are two models of message queuing ： The queue model and Release / Subscription model . RabbitMQ Queue model is adopted , RocketMQ and Kafka Adopt release / Subscription model .

3.1、 The queue model

The producer sends a message to a queue , A queue can store messages from multiple producers , A queue can also have multiple consumers , But there is a competitive relationship between consumers , That is, each message can only be consumed by one consumer .

3.2、 Release / Subscription model

Release / The subscription model sends messages to a Topic That is, in the subject , All subscribed to this Topic All subscribers can consume this message . Release / The subscription model is compatible with the queue model , That is, when there is only one consumer, it is basically consistent with the queue model .

The following content is based on Publishing / Subscription model

4、 Common terms

• Producer : The party that sends the production message is called the producer
• Consumer : The party receiving the consumption information is called the consumer
• Broker : Message queuing server

Message from Producer Sent to Broker , Broker Store messages locally , then Consumer from Broker Pull the news , perhaps Broker Push message to Consumer , Last consumption .

To improve concurrency , Often publish / The subscription model also introduces the concept of queues or partitions . That is, the message is sent to a queue or partition under a topic . RocketMQ Call the queue , Kafka It's called zoning , The essence is the same .

For example, there is... Under a topic 5 A queue , Then the concurrency of this topic is increased to 5 , At the same time, there can be 5 Multiple consumers consume messages on this topic in parallel . Generally, polling or key hash Strategies such as remainder are used to allocate messages of the same subject to different queues .

The corresponding consumers generally have the concept of group Consumer Group , That is, consumers belong to a certain consumption group . A message will be sent to multiple consumer groups that subscribe to this topic .

Suppose there are two consumption groups, namely Group 1 and Group 2 , They're all subscribed to Topic-a . At this time, a message is sent to Topic-a , Then both consumer groups can receive this message .

Then the message is actually written to Topic In a queue , A consumer in the consumption group consumes a queue of messages .

Physically, in addition to copies , A message in Broker There will only be one in , Each consumer group will have its own offset That is, the consumption point is used to identify the consumption location . The news before the consumption point indicates that it has been consumed . Of course, this offset It's queue level . Each consumer group maintains a subscription Topic Of each queue under offset .

Take a look at the picture, it should be very clear .

5、 How to ensure that messages are not lost

There are three stages , Namely Production news 、 Store and consume messages . We start from these three stages to ensure that messages will not be lost . Ensuring the reliability of messages requires Three parties cooperate .

• producer Need to deal with Broker Response , Use retry in case of error 、 Alarm and other means .

• Broker Need to control the timing of response , In the case of a single machine, the response is returned after the message swipes the disk , Cluster multiple replicas , That is, if it is sent to two or more copies, the response will be returned .

• consumer You need to return the response to after executing the real business logic Broker .

Be careful : Message reliability is enhanced , Performance drops , Wait for the message to brush the disk 、 Returning after multiple copies are synchronized will affect performance . So it depends on the business , For example, it doesn't matter if one or two logs are lost during log transmission , Therefore, there is no need to wait for the message to brush the disk before responding .

6、 How to handle duplicate messages

In normal business Message repetition is inevitable , So we can only from Another way to solve the problem of duplicate messages .

Idempotent handles duplicate messages ： Idempotent is a mathematical concept , We understand that calling the same interface multiple times with the same parameters is the same as calling the same interface once .

For example, this article SQL

update t1 set money = 150 where id = 1 and money = 100;

How many times money All are 150, This is called idempotent . Therefore, it is necessary to transform the business processing logic , So that the final result will not be affected in the case of repeated messages .

Database data version control ：version Version number control .

update t1 set money = 150 where id = 1 and version = 1; #  After the update  version + 1

7、 How to ensure the order of the message

7.1、 Overall order

If you want to ensure the global order of messages , First, only one producer can go to Topic Send a message , And one Topic There can only be one queue inside （ Partition ）. The consumer must also consume the queue in a single thread . Such messages are globally ordered ！
But generally, we don't need global order , Even synchronization MySQL Binlog You only need to ensure that the single table messages are in order .

7.2、 Partial order

Therefore, most of the ordered requirements are partially ordered , Partially ordered, we can put Topic Internally divided into the number of queues we need , Send messages to a fixed queue through a specific policy , Then each queue corresponds to a single threaded consumer . In this way, some orderly requirements are completed , The message processing efficiency can be improved by the concurrency of the number of queues .

There are several producers in the picture , A producer can also , As long as similar messages are sent to the specified queue .