Introduction to message queuing (MQ)

1.MQ Use scenarios

 

asynchronous communication  

Some businesses don't want or need to process messages immediately . Message queues provide an asynchronous processing mechanism , Allow users to put messages on the queue , But not immediately . Put as many messages as you want in the queue , Then deal with him when necessary .

 

decoupling  

Reduce the strong dependence between projects , Adapt to heterogeneous systems . Predict what needs the project will encounter in the future at the beginning of the project , It's extremely difficult . The message system inserts an implicit 、 Data based interface layer , Both processes need to implement this interface , When the application changes , You can expand or modify the processing on both sides independently , Just make sure they adhere to the same interface constraints .

 

redundancy  

In some cases , The process of processing data can fail . Unless the data is persisted , Otherwise, it will cause loss . Message queues persist data until they are fully processed , In this way, the risk of data loss is avoided . Used by many message queues " Insert - obtain - Delete " In the paradigm , Before deleting a message from the queue , Requires that your processing system explicitly indicate that the message has been processed , This ensures that your data is stored safely until you have finished using it .

 

Extensibility  

Because message queues decouple your processing , So it is easy to increase the frequency of message enqueue and processing , Just add another process . No need to change code 、 There is no need to adjust the parameters . It is convenient for distributed expansion .

 

Overload protection  

In the case of a dramatic increase in traffic , Applications still need to work , But such burst traffic can't be extracted and predicted ; If we think that we can deal with this kind of instantaneous peak visit as the standard to put resources on standby at any time is undoubtedly a huge waste . Using message queues enables critical components to withstand sudden access pressures , It won't crash completely because of a sudden overload of requests .

 

Recoverability  

When a part of the system fails , It doesn't affect the whole system . Message queues reduce the degree of coupling between processes , So even if a process that processes the message fails , Messages enqueued can still be processed after the system recovers .

 

Sequence assurance  

In most usage scenarios , The order in which the data is processed is important . Most message queues are sorted anyway , And it guarantees that the data will be processed in a particular order .

 

buffer  

In any important system , There will be elements that require different processing times . Message queuing helps the most efficient execution of tasks through a buffer layer , This buffer helps to control and optimize the speed of data flow through the system . To adjust the system response time .

 

Data flow processing  

Massive data streams generated by distributed systems , Such as ： Business log 、 Monitoring data 、 User behavior, etc , Collect and summarize these data streams in real time or in batches , Then big data analysis is the necessary technology of the Internet , It is the best choice to complete this kind of data collection through message queuing .

 

2.MQ principle

 

2.1MQ Model

Pub/Sub Publish subscribe （ radio broadcast ）： Use topic As a communication carrier  

 

PTP Point to point : Use queue As a communication carrier  

 

2.2 MQ The composition of

Broker： Message server , As server Provide message core services

 

Producer: Message producer , The sponsor of the business , Responsible for the transmission of production messages to broker

 

Consumer： Message consumer , The processor of the business , In charge of from broker Get the message and do business logic processing

 

Topic: The theme , Unified message gathering place under publish and subscribe mode , Different producers turn to topic Send a message , from MQ The server distributes to different subscribers , Broadcast messages

 

Queue： queue ,PTP In mode , Specific producers to specific queue Send a message , Consumers subscribe to specific queue Finish receiving the specified message

 

Message： Message body , A packet encoded in a fixed format defined by different communication protocols , To encapsulate business data , Realize the transmission of messages

 

2.3 MQ Common agreement

AMQP 

AMQP namely Advanced Message Queuing Protocol, An application-level standard advanced message queue protocol that provides a unified message service , 

Is an open standard for application layer protocols , Designed for message-oriented middleware . The client and message middleware based on this protocol can deliver messages , and  

No clients / Different middleware products , Restrictions on different development languages and other conditions . 

advantage ： reliable 、 Universal

 

MQTT agreement  

MQTT（Message Queuing Telemetry Transport, Message queuing telemetry transmission ） yes IBM Developed an instant messaging Association  

discussion , It may become an important part of the Internet of things . The protocol supports all platforms , It can connect almost all online items with the outside world , 

Used as sensors and actuators （ Such as through Twitter Let the house network ） Communication protocol of . 

advantage ： Simple format 、 Small bandwidth 、 Mobile communication 、PUSH、 Embedded system

 

STOMP agreement  

STOMP（Streaming Text Orientated Message Protocol） Is the streaming text oriented message protocol , It's for MOM(Message Oriented Middleware, Simple text protocol for message oriented middleware design .STOMP Provide an interoperable connection format , Allow clients with any STOMP The message broker （Broker） Interact . 

advantage ： Command mode （ Not topic\queue Pattern ）

 

XMPP agreement  

XMPP（ Extensible message processing field protocol ,Extensible Messaging and Presence Protocol） It's based on extensible markup language （XML） The agreement , More for instant messaging （IM） And online live detection . For quasi instant operation between servers . The core is based on XML streaming , This protocol may eventually allow Internet users to send instant messages to anyone else on the Internet , Even if its operating system and browser are different . 

advantage ： General public 、 Strong compatibility 、 Scalable 、 High safety , but XML The coding format takes up a large amount of bandwidth

 

The other is based on TCP/IP Custom protocol  

Some special frames （ Such as ：redis、kafka、zeroMq etc. ） Not strictly following according to one's own needs MQ standard , It's based on TCP\IP Self encapsulating a set of agreements , Through the network socket Interface for transmission , Realized MQ The function of .

 

2.4 MQ The selection

RabbitMQ 

Use Erlang Write an open source message queue , It supports a lot of agreements ：AMQP,XMPP, SMTP, STOMP, Exactly so , It's very heavyweight , More suitable for enterprise level development . At the same time Broker framework , The core idea is that producers don't send messages directly to queues , Messages are queued in the central queue when they are sent to the client . Routing (Routing), Load balancing (Load balance)、 Data persistence is well supported . More for enterprise level ESB Integrate .

 

ZeroMQ 

also called ØMQ、0MQ、ZMQ, The fastest message queuing system , Dedicated to high throughput 、 Low latency scenario development , It is often used in financial applications , Focus on real-time data communication scenarios .ZMQ Can achieve RabbitMQ Not good at advanced / Complex queues , But developers need to assemble multiple technical frameworks themselves , High development cost . therefore ZeroMQ With a unique non middleware pattern , More like a socket library, You don't need to install and run a message server or middleware , Because your app itself uses ZeroMQ API Complete the role of logical Services . however ZeroMQ Only non persistent queues , If down machine , Data will be lost . Such as ：Twitter Of Storm Use in ZeroMQ As the transmission of data stream . ZeroMQ Sockets are not related to the transport layer ：ZeroMQ Socket defines a unified... For all transport layer protocols API Interface . The default support In process (inproc) , Interprocess (IPC) , multicast ,TCP agreement , Switching between different protocols simply changes the prefix of the connection string . You can switch from local communication between processes to distributed communication at least at any time TCP signal communication .ZeroMQ In the back processing connection establishment , Disconnect and reconnect logic . 

characteristic ：

 

Lockless queue model  

For cross thread interactions （ The client and session） Data exchange channel between pipe, The queue algorithm without lock is adopted CAS; stay pipe There are asynchronous events registered at both ends of , Reading or writing messages to pipe When the , Will automatically trigger read and write events .

 

Algorithms for batch processing  

For bulk messages , We optimized the adaptability , Can receive and send messages in batches .

 

Thread binding under multi-core , need not CPU Switch  

Different from the traditional multithreading mode , Semaphore or critical area , zeroMQ Make full use of the advantages of multi-core , Each core binding runs a worker thread , Avoid multithreading between CPU Switching overhead .

 

ActiveMQ 

Apache Next sub project . Use Java Fully support JMS1.1 and J2EE 1.4 canonical JMS Provider Realization , A small amount of code can efficiently implement advanced application scenarios . Pluggable transport protocol support , such as ：in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports.RabbitMQ、ZeroMQ、ActiveMQ All support common multi language clients C++、Java、.Net,、Python、 Php、 Ruby etc. .

 

Redis 

Use C Language development Key-Value Of NoSQL database , Development and maintenance are very active , Although it is a Key-Value Database storage system , But it supports MQ function , So it can be used as a lightweight queue service . about RabbitMQ and Redis The entry and exit operations of , Each executive 100 Ten thousand times , Every time 10 Ten thousand times record the execution time . The test data is divided into 128Bytes、512Bytes、1K and 10K Four different sizes of data . Experiments show that ： When you join the team , When the data is small Redis The performance is higher than RabbitMQ, And if the data size exceeds 10K,Redis It's too slow to bear ; When the team , No matter the size of the data ,Redis All showed very good performance , and RabbitMQ The out of team performance is much lower than Redis.

 

Kafka 

Apache Next sub project , Use scala Implementation of a high-performance distributed Publish/Subscribe Message queuing system , It has the following characteristics ： Fast persistence ： Through the disk sequential read-write and zero copy mechanism , Can be in O(1) Message persistence under the system overhead of ; 

High throughput ： It can be achieved on a common server 10W/s The throughput rate of ; 

High accumulation ： Support topic The next consumer is offline for a long time , There's a lot of news piling up ; 

A completely distributed system ：Broker、Producer、Consumer All native automatic support distributed , rely on zookeeper Automatic realization of complex equilibrium ; 

Support Hadoop Data is loaded in parallel ： For things like Hadoop The same log data and offline analysis system , But it also requires the limitation of real-time processing , This is a viable solution .