Introduction to several common usage scenarios of message queue

One 、 brief introduction

Message Queuing Middleware is an important component of distributed system , Mainly solve application coupling , Asynchronous messaging , Flow cutting . Achieve high performance 、 High availability 、 Scalable and ultimately consistent architecture . Many message queues are ActiveMQ、RabbitMQ、ZeroMQ、Kafka、MetaMQ、RocketMQ.

Two 、 Message queuing application scenario

The following describes the common use scenarios of message queuing in practical applications ： Asynchronous processing , The application of decoupling , Four scenarios of traffic cutting and message communication .

1、 Asynchronous processing
The scene that ： After user registration , Need to send registration email and SMS . There are two traditional approaches ： Serial mode and parallel mode .

Serial mode ： After successfully writing the registration information to the database , Send registration email , Resend registration SMS . After all the above three tasks are completed , Return to customer .

Parallel mode ： After successfully writing the registration information to the database , Send registration email at the same time , Send registration SMS . After the above three tasks are completed , Return to the client . The difference with serial is , Parallel processing can improve processing time

Summary ： As described in the above case , Performance of traditional way system （ Concurrency , throughput , response time ） There will be bottlenecks. . How to solve this problem ？

Import message queue , Will not be a required business logic , Asynchronous processing . The reconstructed structure is as follows ：

As agreed above , The response time of the user is equal to the time when the registration information is written to the database , That is to say 50 millisecond . Registered mail , After SMS is written to message queue , Go straight back to , So writing to message queues is fast , Basically negligible , So the response time of the user may be 50 millisecond . So after the architecture changes , System throughput increased to per second 20QPS. Better than serial 3 times , Twice as much as parallel ！

2、 The application of decoupling
The scene that ： After the user orders , Order system needs to inform inventory system . The traditional way is , Interface between order system and inventory system . Here's the picture ：

Disadvantages of traditional models ：
If the inventory system is not accessible , Order inventory reduction will fail , Which leads to order failure , Coupling of order system and inventory system .

How to solve the above problems ？ Scheme after introducing application message queue , Here's the picture ：

Order system ： After the user orders , Order system completes persistent processing , Write message to message queue , Return to user's order successfully placed

inventory system ： Subscribe to the message of the order , Use pull / Push way , Get order information , Inventory system according to order information , Inventory operation

If ： The inventory system cannot be used normally when placing an order . It does not affect normal order , Because after placing the order , When the order system writes to the message queue, it no longer cares about other subsequent operations . Realize the application decoupling of order system and inventory system .

3、 Flow cutting
Traffic cutting is also a common scenario in message queuing , Generally, it is widely used in seckill or group robbery activities ！

Application scenarios ： Seckill activity , Generally, it will be caused by excessive flow , Resulting in a surge in flow , Hang up . To solve this problem , In general, it is necessary to join the message queue in the front end of the application .

Number of people who can control the activity , It can alleviate the application of high flow collapse in a short time .

User's request , After server receives , Write message queue first . If the message queue length exceeds the maximum number , Discard the user request or jump to the error page directly .

Seckill service according to the request information in the message queue , Follow up .

4、 Log processing
Log processing refers to the use of message queuing in log processing , such as Kafka Application , Solve the problem of a large number of log transfers . The architecture is simplified as follows ：

Log collection client , Responsible for log data collection , Timed write to write Kafka queue ;Kafka Message queue , Responsible for receiving log data , Store and forward ; Log processing application ： Subscribe and consume kafka Log data in queue .

Here is Sina kafka Log processing application case ：

Kafka： Message queue to receive user logs ;
Logstash： Log parsing , Unification JSON Output to Elasticsearch;
Elasticsearch： Core technology of real time log analysis service , One schemaless, Real time data storage service , adopt index Organization data , Powerful search and statistics ;
Kibana： be based on Elasticsearch Data visualization component of , Superior data visualization ability is the choice of many companies ELK stack Important reasons .

5、 Message communication
Message communication means , Message queuing usually has an efficient communication mechanism built in , So it can also be used in pure message communication . For example, implementing point-to-point message queuing , Or chat rooms, etc .

Point to point communication ：

client A And the client B Use the same queue , Message communication .

Chat room communication ：
client A, client B, client N Subscribe to the same topic , Publish and receive messages . Achieve a chat like effect .

These are actually two message modes of message queuing , Point to point or publish subscription mode . Model is schematic , For reference .

3、 ... and 、 Message middleware example

1、 Electricity supplier system

Message queuing with high availability , Persistent message middleware . such as Active MQ,Rabbit MQ,Rocket Mq.

After the application completes the backbone logic processing , Write message queue . Whether the message is sent successfully can open the message confirmation mode .（ After message queue returns message receiving success status , Application return , This ensures message integrity ）;

Expansion process （ texting , Distribution processing ） Subscription queue message . Get and process messages by push or pull ;

Message will be decoupled , Brings data consistency issues , Can be resolved in a final consistent way . For example, the master data is written to the database , Extended application based on message queue , Combined with the database mode to realize the follow-up processing based on message queue ;

2、 Log collection system

It is divided into Zookeeper Registry Center , Log collection client ,Kafka Clusters and Storm colony （OtherApp） Four-part composition .