What is message queuing?

Catalog

One 、 What is message queuing

Two 、 Message queuing application scenario

1、 Asynchronous processing

2、 The application of decoupling

3、 Flow cutting

4、 Log processing

5、 Message communication

3、 ... and 、 Message middleware case

1、 Example of e-commerce system

2、 Example of log collection system

Four 、MQ Advantages and disadvantages of

---

One 、 What is message queuing

         Message queue , Has become a “ Message middleware ”

         news （Message） It refers to the data transmitted between applications . The message can be very simple , For example, it only contains text strings , It can be more complicated , May contain embedded objects .

         Message queue （Message Queue） It's a way of communication between applications , The message can be returned to , The message system ensures the reliable delivery of messages . The information publisher only publishes the information to MQ It doesn't matter who gets it , The message consumer only from MQ Get the message no matter who publishes it . In this way, neither the publisher nor the user needs to know the existence of each other .

         Message queuing is an asynchronous cooperation mechanism between applications , At the same time, message queue middleware is an important component in distributed system , Mainly solve application coupling , Asynchronous messaging , Flow cutting . Achieve high performance , High availability , Scalable and ultimately consistent architecture . It is an indispensable middleware for large distributed system .

Two 、 Message queuing application scenario

Respectively introduced in ： Asynchronous processing , The application of decoupling , Traffic cutting and message communication Four scenarios .

1、 Asynchronous processing

         The scene that ： After user registration , Need to send registration email and SMS . There are two traditional approaches 1. Serial mode ;2. Parallel mode .

（1） Serial mode ： After successfully writing the registration information to the database , Send registration email , Resend registration SMS . After all the above three tasks are completed , Return to the client .

（2） Parallel mode ： After successfully writing the registration information to the database , Send registration email at the same time , Send registration SMS . After the above three tasks are completed , Return to the client .

1） The difference between serial and parallel ？

Suppose three business nodes each use 50 Milli second , No consideration of network and other expenses , The time of serial mode is 150 millisecond , Parallel time may be 100 millisecond .

because CPU The number of requests processed in unit time is certain , hypothesis CPU1 Throughput per second is 100 Time . Serial mode 1 Seconds CPU The number of requests that can be processed is 7 Time （1000/150）. The number of requests processed in parallel is 10 Time （1000/100）.

Summary ： Parallel processing can improve processing time .

2） As described in the above case , Performance of traditional way system （ Concurrency , throughput , response time ） There will be bottlenecks. . How to solve this problem ？

Import message queue , Will not be a required business logic , Asynchronous processing . The reconstructed structure is as follows ：

As agreed above , The response time of the user is equal to the time when the registration information is written to the database , That is to say 50 millisecond . Registered mail , After SMS is written to message queue , Go straight back to , So writing to message queues is fast , Basically negligible , So the response time of the user may be 50 millisecond . So after the architecture changes , System throughput increased to per second 20 QPS. Better than serial 3 times , Twice as much as parallel .

Summary ： Import message queue Greatly improve the system performance of traditional methods

2、 The application of decoupling

The scene that ： After the user orders , Order system needs to inform inventory system . The traditional way is , Interface between order system and inventory system . Here's the picture ：

 

Disadvantages of traditional models ：

1）  If the inventory system is not accessible , Order inventory reduction will fail , Which leads to order failure ;

2）  Coupling of order system and inventory system ;

How to solve the above problems ？ Scheme after introducing application message queue , Here's the picture ：

• Order system ： After the user orders , Order system completes persistent processing , Write message to message queue , Return to user's order successfully placed .
• inventory system ： Subscribe to the message of the order , Use pull / Push way , Get order information , Inventory system according to order information , Inventory operation .
• If ： The inventory system cannot be used normally when placing an order . It does not affect normal order , Because after placing the order , When the order system writes to the message queue, it no longer cares about other subsequent operations . Realize the application decoupling of order system and inventory system .

3、 Flow cutting

Traffic cutting is also a common scenario in message queuing , Generally, it is widely used in seckill or group robbery activities .

Application scenarios ：

Seckill activity , Generally, it will be caused by excessive flow , Resulting in a surge in flow , Hang up . To solve this problem , In general, it is necessary to join the message queue in the front end of the application .

1） Number of people who can control the activity ;

2） It can alleviate the application of high flow collapse in a short time ;

 

Processing mode ：

1） User's request , After server receives , Write message queue first . If the message queue length exceeds the maximum number , Discard the user request or jump to the error page directly ;

2） Seckill service according to the request information in the message queue , Follow up .

4、 Log processing

Log processing refers to the use of message queuing in log processing , such as Kafka Application , Solve the problem of a large number of log transfers . The architecture is simplified as follows ：

• Log collection client , Responsible for log data collection , Timed write to write Kafka queue ;
• Kafka Message queue , Responsible for receiving log data , Store and forward ;
• Log processing application ： Subscribe and consume kafka Log data in queue ;

Here is Sina kafka Log processing application case ：

(1) Kafka： Message queue to receive user logs .

(2) Logstash： Log parsing , Unification JSON Output to Elasticsearch.

(3) Elasticsearch： Core technology of real time log analysis service , One schemaless, Real time data storage service , adopt index Organization data , Powerful search and statistics .

(4) Kibana： be based on Elasticsearch Data visualization component of , Superior data visualization ability is the choice of many companies ELK stack Important reasons .

5、 Message communication

Message communication means , Message queuing usually has an efficient communication mechanism built in , So it can also be used in pure message communication . For example, implementing point-to-point message queuing , Or chat rooms, etc .

Point to point communication ：

client A And the client B Use the same queue , Message communication .

Chat room communication ：

 

client A, client B, client N Subscribe to the same topic , Publish and receive messages , Achieve a chat like effect .

3、 ... and 、 Message middleware case

1、 Example of e-commerce system

Message queuing with high availability , Persistent message middleware . such as Active MQ,Rabbit MQ,Rocket Mq ,kafka

（1） After the application completes the backbone logic processing , Write message queue . Whether the message is sent successfully can open the message confirmation mode .（ After message queue returns message receiving success status , Application return , This ensures message integrity ）

（2） Expansion process （ texting , Distribution processing ） Subscription queue message . Get and process messages by push or pull .

（3） Message will be decoupled , Brings data consistency issues , Can be resolved in a final consistent way . For example, the master data is written to the database , Extended application based on message queue , Combined with the database mode to realize the follow-up processing based on message queue .

2、 Example of log collection system

Message queuing with high availability , Persistent message middleware . such as Active MQ,Rabbit MQ,Rocket Mq ,kafka

（1） After the application completes the backbone logic processing , Write message queue . Whether the message is sent successfully can open the message confirmation mode .（ After message queue returns message receiving success status , Application return , This ensures message integrity ）

（2） Expansion process （ texting , Distribution processing ） Subscription queue message . Get and process messages by push or pull .

（3） Message will be decoupled , Brings data consistency issues , Can be resolved in a final consistent way . For example, the master data is written to the database , Extended application based on message queue , Combined with the database mode to realize the follow-up processing based on message queue .

Four 、MQ Advantages and disadvantages of

advantage ： decoupling 、 Peak shaving 、 Data dissemination

Disadvantages include the following ：

1） Reduced system availability

• • The more external dependencies the system introduces , The less stable the system is . once MQ Downtime , It will have an impact on the business .
  • How to ensure MQ High availability ？

2） The complexity of the system increases

• • MQ And the complexity of the system is greatly increased , Before, there were synchronous remote calls between systems , Now through MQ Make asynchronous calls .
  • How to ensure that messages are not consumed repeatedly ？ How to deal with the loss of information ？ Then ensure the order of message delivery ？

3） Consistency issues

A The system processes the business , adopt MQ to B、C、D Three systems send message data , If B System 、C The system processing is successful ,D System processing failed . How to ensure the consistency of message data processing ？