Introduction to message queuing

Why use message queuing ?

 

analysis : A person using message queuing , I don't know why , It 's a little awkward . No review , It's easy to be asked , And then it's bullshit .

answer : This problem , Let's just answer three main application scenarios ( There is no denying that there are other , But only three main ones ), The following six words : decoupling 、 asynchronous 、 Peak shaving

 

(1) decoupling

The traditional model :

 

Disadvantages of traditional models ：

 

Coupling between systems is too strong , As shown in the figure above , System A Call the system directly in the code B And system C Code for , If future D System access , System A You also need to modify the code , Too much trouble ！

Middleware mode :

 

Advantages of middleware pattern ：

 

Write message to message queue , The system that needs the message subscribes itself from the message queue , Thus the system A No modification required .

(2) asynchronous

The traditional model :

 

Disadvantages of traditional models ：

 

Some unnecessary business logic runs synchronously , Too time consuming .

Middleware mode :

 

Advantages of middleware pattern ：

 

Write message to message queue , Unnecessary business logic runs asynchronously , Faster response

(3) Peak shaving

The traditional model

 

Disadvantages of traditional models ：

 

When the concurrency is large , All requests are directly connected to the database , Cause database connection exception

Middleware mode :

 

Advantages of middleware pattern ：

 

System A Slowly follow the concurrency that the database can handle , Slowly pull messages from the message queue . In production , This short peak backlog is allowed .

2、 What are the disadvantages of using message queuing ?

analysis : One is used MQ Project , If you don't even think about it , Just put MQ The introduction has gone. , That brings risks to your project . We introduced a technology , We should have a full understanding of the disadvantages of this technology , To prevent . Remember , Don't dig holes for the company ！

answer : It's easy to answer , Answer from the following two perspectives

 

Reduced system availability : You want to. , Other systems would have to work well , Then your system is normal . Now you have to add a message queue , The message queue hangs , Your system is not ha ha . therefore , Reduced system availability

Increased system complexity : There are many aspects to consider , Like consistency 、 How to ensure that messages are not consumed repeatedly , How to ensure reliable transmission of messages . therefore , More to think about , System complexity increases .

however , Shall we use it or not .

 

3、 How to select message queue ?

But first, let me say , Bloggers only ActiveMQ,RabbitMQ,RocketMQ,Kafka, What to ZeroMQ Others, such as MQ Nothing to understand , So it can only be based on these four MQ Give an answer .

analysis : Now that it's used in the project MQ, It must be popular in the industry in advance MQ Conduct research , If every kind of MQ I didn't understand the advantages and disadvantages of , Just pat your head as you like , Used some kind of MQ, Or dig a hole for the project . If the interviewer asks :" Why do you use this MQ？." You answer directly " Decided by leaders ." That's the answer LOW 了 . Or that sentence , Don't dig holes for the company .

answer : First , Let's go first. ActiveMQ Our community , Look at this. MQ Update frequency of :

 

 

Apache ActiveMQ 5.15.3 Release

 

Christopher L. Shannon posted on Feb 12, 2018

 

Apache ActiveMQ 5.15.2 Released

 

Christopher L. Shannon posted on Oct 23, 2017

 

Apache ActiveMQ 5.15.0 Released

 

Christopher L. Shannon posted on Jul 06, 2017

 

Omit the following records

 

...

We can see that ,ActiveMq It's only released once a few months , It is said that the research focus is on their next generation products Apollo.

Next , Let's go again. RabbitMQ Take a look at our community ,RabbitMQ Update frequency of

 

 

RabbitMQ 3.7.3 release 30 January 2018

 

RabbitMQ 3.6.15 release 17 January 2018

 

RabbitMQ 3.7.2 release23 December 2017

 

RabbitMQ 3.7.1 release21 December 2017

 

Omit the following records

 

...

We can see that ,RabbitMQ Version release ratio ActiveMq Frequent . as for RocketMQ and kafka I won 't show you , In short, it is also better than ActiveMQ More active . details , Self access .

Another performance comparison table

 

characteristic ActiveMQ RabbitMQ RocketMQ kafka

development language java erlang java scala

Single machine throughput All level All level 10 All level 10 All level

timeliness ms level us level ms level ms Level of less than

Usability high ( Master slave architecture ) high ( Master slave architecture ) Very high ( Distributed architecture ) Very high ( Distributed architecture )

features Mature products , Applied in many companies ; There are many documents ; Good support for various protocols be based on erlang Development , So the concurrency is very strong , Extremely good performance , The delay is very low ; Rich management interface MQ The functions are relatively complete , Good scalability Only major MQ function , Like some message query , Message backtracking and other functions are not provided , After all, it's for big data , Widely used in big data field .

Combining the above materials, we can get the following two points :

(1) Small and medium software companies , Recommended election RabbitMQ. One side ,erlang Languages are inherently highly concurrent , And his management interface is very convenient to use . As the saying goes , Cheng Yi Xiao He , Xiao He also defeated ！ And here's what's wrong with him , although RabbitMQ It's open source. , However, there are several customized developments in China erlang What about the programmers ？ Fortunately ,RabbitMQ 's community is very active , It can solve the problems encountered in the development process bug, This is very important for small and medium-sized companies . Don't consider rocketmq and kafka The reason is that , On the one hand, small and medium-sized software companies are inferior to Internet companies , The amount of data is not that large , Select message middleware , We should first choose the one with complete functions , therefore kafka exclude . Don't consider rocketmq The reason is that ,rocketmq It's made by Ali , If Ali gives up maintenance rocketmq, Small and medium-sized companies usually don't have people to do it rocketmq Customized development of , Therefore, it is not recommended .

(2) Large software companies , According to the specific use rocketMq and kafka Choose between two . One side , Large software companies , Enough funds to build a distributed environment , It also has enough data . in the light of rocketMQ, Large software companies can also pull out people rocketMQ Customized development , After all, China has the ability to change JAVA Source person , Still quite a lot . as for kafka, Select according to business scenario , If there is log collection function , Must be the first choice kafka 了 . Which should I choose , See the use scenario .

 

4、 How to ensure that message queues are highly available ？

analysis : On the second point , After message queuing is introduced , Reduced availability of the system . In production , No one uses message queuing in stand-alone mode . therefore , As a qualified programmer , You should have a deep understanding of the high availability of message queuing . If at the time of the interview , Interviewer asked , How to ensure high availability of your message middleware ？ Your answer just shows that you can only subscribe to and publish messages , The interviewer will wonder if you're just playing on your own , It's not used in production . Please think about it , Will think , Thinking programmer .

answer : This problem , In fact, we need to have a deep understanding of the clustering mode of message queuing , Good answer. .

With rcoketMQ For example , There are so many of them master Pattern 、 many master many slave Asynchronous replication mode 、 many master many slave Synchronous double write mode . many master many slave Pattern deployment architecture ( Internet search , Steal laziness , Too lazy to paint ):

image

In fact, the blogger saw this picture at the first sight , Feel and kafka As if , It's just NameServer colony , stay kafka Medium used zookeeper Instead of , It's all for preservation and discovery master and slave With . The communication process is as follows :

Producer And NameServer One of the nodes in the cluster （ Random selection ） Establish long connection , On a regular basis from NameServer obtain Topic Routing information , And provide Topic Service Broker Master Establish long connection , And timing to Broker Send a heartbeat .Producer Messages can only be sent to Broker master, however Consumer Is not the same , It also provides Topic Service Master and Slave Establish long connection , It can be done from Broker Master Subscribe to news , You can also get it from Broker Slave Subscribe to news .

that kafka Well , For comparison kafka Extension and supplement structure of ( It's also looking for it. , Too lazy to paint )

image

As shown in the figure above , A typical Kafka The cluster contains several Producer（ It can be web Front end generated Page View, Or server logs , System CPU、Memory etc. ）, A number of broker（Kafka Support horizontal scaling , commonly broker The more the number of , Higher cluster throughput ）, A number of Consumer Group, And one. Zookeeper colony .Kafka adopt Zookeeper Manage cluster configuration , The election leader, And in Consumer Group When changes occur rebalance.Producer Use push The pattern publishes the message to broker,Consumer Use pull The pattern is from broker Subscribe and consume messages .

as for rabbitMQ, There are also general cluster and image cluster modes , Get to know for yourself , Relatively simple , Two hours to understand .

requirement , When answering highly available questions , Should be able to draw their own logically and clearly MQ Cluster architecture or clear description .

 

5、 How to ensure that messages are not consumed repeatedly ？

analysis : In fact, another way to ask this question is , How to guarantee the idempotence of message queue ? This problem can be regarded as the basic problem in the field of message queuing . In other words , Is to examine your design ability , The answer to this question can be based on specific business scenarios , There is no fixed answer .

answer : First of all, why does it cause repeated consumption ?

   In fact, no matter what kind of message queue , The causes of repeated consumption are similar . Under normal circumstances , When consumers consume news , After consumption , A confirmation message will be sent to the message queue , The message queue knows that the message has been consumed , The message is removed from the message queue . But different message queues send different forms of confirmation information , for example RabbitMQ Send one ACK Confirmation message ,RocketMQ Is to return a CONSUME_SUCCESS Sign of success ,kafka There's actually a offset The concept of , Just briefly ( If you don't understand , Go out and find one kafka Introduction to mastery ), Every message has one offset,kafka After consuming the news , Need to submit offset, Let message queuing know that it has consumed . That's what causes repeat consumption ?, It's because of network transmission and so on , Confirm message not delivered to message queue , The message queue does not know that it has consumed the message , Distribute the message to other consumers again .

   How to solve ? This question is divided into the following points according to the business scenario

  (1) such as , You've got this message for the database insert operation . That's easy , Make a unique primary key for this message , So even if there is a situation of repeated consumption , Will lead to primary key conflict , Avoid dirty data in database .

  (2) Another example , You get the news redis Of set The operation of , That's easy , No need to solve , Because you don't care set The results are the same several times ,set Operation is idempotent .

  (3) If the above two conditions are not good , Great trick . Prepare a third-party media , To make consumption records . With redis For example , Assign a global to the message id, As long as the message has been consumed , take <id,message> With K-V Form write redis. Before consumers start to spend , Go first redis Check whether there is consumption record in