[suggestions collection] common usage scenarios of message queue

1.  system decoupling

Look at this scene .A The system sends data to BCD Three systems , Sent via interface call . If E The system also needs this data ？ Then if C The system is not needed now ？A The head of the system almost collapsed ......

In this scenario ,A The system is heavily coupled to all sorts of other messy systems ,A The system produces a key piece of data , Many systems need A The system sends this data .A The system has to think about it all the time BCDE What if all four systems fail ？ Do you want to resend , Do you want to save the message ？ My hair is white ！

If you use MQ,A The system generates a piece of data , Send to MQ Go inside , Which system needs the data to go by itself MQ In consumption . If the new system requires data , Directly from MQ It's OK to spend in ; If a system doesn't need this data , Just cancel it. Yes MQ The consumption of news can be . So down ,A The system doesn't even have to think about who it's sending data to , There is no need to maintain this code , There is no need to consider whether the call was successful 、 Time out, failure, etc .

  summary ： Through one MQ,Pub/Sub Publish subscribe message such a model ,A The system is completely decoupled from the other systems .

therefore , We need to consider whether there are similar scenarios in our business system , It's a system or a module , Multiple systems or modules called , Calls to each other are complex , It's hard to maintain . But in fact, this call does not need to directly synchronize the calling interface , If you use MQ Decouple it asynchronously , It's OK, too , You need to think about it in your project , Is it possible to use this MQ To decouple the system .

---

2. Asynchronous processing

Let's do another scene ,A The system receives a request , You need to write your own library , Still need to be in BCD Three system write Libraries , You need to write your own library 3ms,BCD Three systems respectively write library requirements 300ms、450ms、200ms. The total delay of the final request is 3 + 300 + 450 + 200 = 953ms, near 1s, What's the user feeling , Slow to death... Slow to death . The user initiates the request through the browser , Waiting for a 1s, This is almost unacceptable .

General Internet enterprises , Direct action to the user , The general requirement is that every request must be in 200 ms Finish within , It is almost insensitive to the user .

If Use MQ, that A The system sends 3 Message to the MQ In line , If it takes time 5ms,A The system goes from accepting a request to returning a response to the user , The total time is 3 + 5 = 8ms, For users , It feels like a button ,8ms And then he went straight back , Shuang ！ Well done website , So fast ！

---

3.  Traffic peak clipping

Every day 0:00 To 12:00,A The system is calm , The number of concurrent requests per second is 50 individual . It turns out to be 12:00 ~ 13:00 , The number of concurrent requests per second suddenly increases to 5k+ strip . But the system is directly based MySQL Of , A flood of requests poured in MySQL, For every second MySQL To perform some 5k strip SQL.

General MySQL, Carry it to every second 2k Three requests is enough , If you request to 5k Words , Maybe just put MySQL To have killed , Causing the system to crash , Users can no longer use the system .

But the rush is over , By the afternoon , It's a low peak , Maybe it's just 1w At the same time on the site , The number of requests per second is probably the same 50 A request , There is almost no pressure on the whole system .

If you use MQ, Per second 5k Requests to write to MQ,A Maximum processing per second 2k A request , because MySQL At most per second 2k individual .A System from MQ Slowly pull the request in , Pull every second 2k A request , Do not exceed the maximum number of requests per second you can handle ok, So down , Even at rush hour ,A The system will never fail . and MQ Every second 5k A request to come in , Just 2k A request to go out , The result is a midday rush （1 Hours ）, There could be hundreds of thousands or even millions of requests MQ in .

This short peak backlog is ok Of , Because after the rush hour , Every seconds 50 A request to enter MQ, however A The system is still going to be per second 2k The speed of the request in processing . So , Just after the rush hour ,A The system will quickly clear the backlog of messages .

---

4.  Log processing

Large e-commerce website （ TaoBao 、 JD.COM 、 Gome 、 Suning ...）、App（ Tiktok 、 Meituan 、 Didi et al ） Need to analyze user behavior , We should find the user's preferences and activity according to the user's access behavior , You need to collect a lot of user access information on the page .

---

Advantages and disadvantages of message queuing

The advantages have been mentioned above , Namely There are corresponding advantages in special scenes .

The disadvantages are as follows ：

• Reduced system availability

  The more external dependencies the system introduces , The easier it is to hang up . You are A system call BCD The interface of the three systems is good ,ABCD Four systems are good , No problem , You add another one. MQ Come in , In case MQ How to deal with it ？MQ A hang , System breakdown , How to ensure high availability of message queues ？

• The complexity of the system increases

  Add one MQ Come in , How can we ensure that messages are not consumed repeatedly ？ How to ensure the sequence of message delivery ？

• Consistency issues

  A After the system finishes processing, it returns to success directly , People think your request is a success ; But the problem is , If BCD Three systems ,BD Two systems have successfully written to the library , result C The system failed to write to the library , What do I do ？ Your data is inconsistent .

  So message queuing is actually a very complex architecture , There are many advantages to introducing it , But it also has to do a variety of additional technical solutions and architecture to avoid the disadvantages , When it's done , You'll find that , Mama ah , System complexity increased by an order of magnitude , Maybe it's complicated 10 times . But the moment of truth , use , Still have to use .

---

Kafka、ActiveMQ、RabbitMQ、RocketMQ  Advantages and disadvantages

Message queue ActiveMQ 、RocketMQ 、RabbitMQ and Kafka How to choose ？

Sum up , After all the comparisons , Here are some suggestions ：

General business systems to be introduced MQ, At first, everyone used ActiveMQ, But people don't really use it much anymore , The large throughput scenario is not validated , The community is not very active either , So let's forget it , I personally don't recommend it .

Later, people began to use RabbitMQ, But it is erlang Language prevents a lot of Java Engineers to study it and master it , For the company , Almost out of control , But they are open source , More stable support , It's also very active .

But now more and more companies will use it RocketMQ, It's really nice , After all, it's made by Ali , But the community may be at risk of a sudden meltdown （ at present RocketMQ Donated to Apache, but GitHub In fact, the level of activity is not high ） Have absolute confidence in oneself company technical strength , Recommend to use RocketMQ, Otherwise go back to old honest practical RabbitMQ Well , They have an active open source community , Absolutely not yellow .

In terms of technology selection ：

Small and medium-sized companies , The technical strength is relatively average , The technical challenges are not particularly high , use RabbitMQ It's a good choice ;

Big companies , Infrastructure research and development strength is strong , use RocketMQ It's a good choice ;

If it is Big data In real time 、 Log collection and other scenarios , use Kafka It's industry standard , Absolutely no problem , The community is very active , Absolutely not yellow , Moreover, it is the DE facto norm in this field almost all over the world .