Message Oriented Middleware (I) MQ explanation and comparison of four MQS

One 、 Knowledge of message middleware

1、 summary

Message queuing has gradually become an enterprise IT The core means of system internal communication . It has low coupling 、 Reliable delivery 、 radio broadcast 、 flow control 、 Finally, a series of functions such as consistency , Become asynchronous RPC One of the main means of . There are many mainstream message middleware in the market today , Like the old brand ActiveMQ、RabbitMQ, Hot Kafka, Alibaba develops on its own RocketMQ etc. .

 

2、 The composition of message middleware

      2.1 Broker

Message server , As server Provide message core services

      2.2 Producer

Message producer , The sponsor of the business , Responsible for the transmission of production messages to broker,

      2.3 Consumer

Message consumer , The processor of the business , In charge of from broker Get the message and do business logic processing

      2.4 Topic

The theme , Unified message gathering place under publish and subscribe mode , Different producers turn to topic Send a message , from MQ The server distributes to different subscribers , Realize the message of        radio broadcast

      2.5 Queue

queue ,PTP In mode , Specific producers to specific queue Send a message , Consumers subscribe to specific queue Finish receiving the specified message

      2.6 Message

Message body , A packet encoded in a fixed format defined by different communication protocols , To encapsulate business data , Realize the transmission of messages

 

3 Message middleware pattern classification

      3.1 Point to point

PTP Point to point : Use queue As a communication carrier  
 
explain ： 
Message producers produce messages sent to queue in , Then the message consumer from queue To retrieve and consume messages . 
After the news was consumed ,queue No more storage in , So it's impossible for news consumers to consume the information that has been consumed . Queue Support for multiple consumers , But for a message , Only one consumer can consume .

3.2 Release / subscribe

Pub/Sub Publish subscribe （ radio broadcast ）： Use topic As a communication carrier  
 
explain ： 
Message producer （ Release ） Publish a message to topic in , There are multiple message consumers at the same time （ subscribe ） Consume the news . It's different from point-to-point , Publish to topic Will be consumed by all subscribers .

queue Load balancing is realized , take producer The production message is sent to the message queue , Consumption by multiple consumers . But a message can only be accepted by one consumer , When no consumers are available , This message will be saved until a consumer is available . 
topic Implemented publishing and subscription , When you post a message , All subscriptions to this topic All of our services can get the news , So from 1 To N Every subscriber can get a copy of a message .

 

4 Advantages of message middleware

      4.1 system decoupling

There is no direct call relationship between interactive systems , Just by message transmission , So the system is not very invasive , Low coupling .

      4.2 Improve system response time

For example, the original set of logic , Completing the payment may involve modifying the order status first 、 Calculate member points 、 Only by notifying the logistics and distribution can the logic be completed ; adopt MQ Architecture design , It will make the emergency important （ Need immediate response ） The business is put into the calling method , Use message queuing with low response requirements , Put it in MQ In line , For consumers to deal with .

      4.3 Provide services for big data processing architecture

Through message as integration , In the context of big data , Message queuing is also integrated with the real-time processing architecture , Provide performance support for data processing .

      4.4 Java Message service ——JMS

Java Message service （Java Message Service,JMS） The API is a Java About message-oriented middleware in the platform （MOM） Of API, Used between two applications , Or send a message in a distributed system , Asynchronous communication . 
JMS Medium P2P and Pub/Sub Message schema ： Point to point （point to point, queue） And publish subscriptions （publish/subscribe,topic） It was originally made by JMS Defined . The main difference between the two modes or the problem to be solved is whether the messages sent to the queue can be consumed repeatedly ( Subscribe more ).

 

5 Message middleware application scenario

       5.1 asynchronous communication

Some businesses don't want or need to process messages immediately . Message queues provide an asynchronous processing mechanism , Allows the user to queue a message , But not immediately . Put as many messages as you want into the queue , Then deal with them as needed .

      5.2 decoupling

Reduce the strong dependence between projects , Adapt to heterogeneous systems . Predict what needs the project will encounter in the future at the beginning of the project , It's extremely difficult . The message system inserts an implicit 、 Data based interface layer , Both processes need to implement this interface , When the application changes , You can expand or modify the processing on both sides independently , Just make sure they adhere to the same interface constraints .

      5.3 redundancy

In some cases , The process of processing data can fail . Unless the data is persisted , Otherwise, it will cause loss . Message queues persist data until they are fully processed , In this way, the risk of data loss is avoided . Used by many message queues ” Insert - obtain - Delete ” In the paradigm , Before deleting a message from the queue , Requires that your processing system explicitly indicate that the message has been processed , This ensures that your data is stored safely until you have finished using it .

      5.4 Extensibility

Because message queues decouple your processing , So it is easy to increase the frequency of message enqueue and processing , Just add another process . No need to change code 、 There is no need to adjust the parameters . It is convenient for distributed expansion .

      5.5 Overload protection

In the case of a dramatic increase in traffic , Applications still need to work , But such burst traffic can't be extracted and predicted ; If we think that we can deal with this kind of instantaneous peak visit as the standard to put resources on standby at any time is undoubtedly a huge waste . Using message queues enables critical components to withstand sudden access pressures , It won't crash completely because of a sudden overload of requests .

      5.6 Recoverability

When a part of the system fails , It doesn't affect the whole system . Message queues reduce the degree of coupling between processes , So even if a process that processes the message fails , Messages enqueued can still be processed after the system recovers .

      5.7 Sequence assurance

In most usage scenarios , The order in which the data is processed is important . Most message queues are sorted anyway , And it guarantees that the data will be processed in a particular order .

      5.8 buffer

In any important system , There will be elements that require different processing times . Message queuing helps the most efficient execution of tasks through a buffer layer , This buffer helps to control and optimize the speed of data flow through the system . To adjust the system response time .

      5.9 Data flow processing

Massive data streams generated by distributed systems , Such as ： Business log 、 Monitoring data 、 User behavior, etc , Collect and summarize these data streams in real time or in batches , Then big data analysis is the necessary technology of the Internet , It is the best choice to complete this kind of data collection through message queuing .

 

6 Common protocols for message middleware

      6.1 AMQP agreement

AMQP namely Advanced Message Queuing Protocol, An application-level standard advanced message queue protocol that provides a unified message service , Is an open standard for application layer protocols , Designed for message-oriented middleware . The client and message middleware based on this protocol can deliver messages , Not by the client / Different middleware products , Restrictions on different development languages and other conditions . 
advantage ： reliable 、 Universal

      6.2 MQTT agreement

MQTT（Message Queuing Telemetry Transport, Message queuing telemetry transmission ） yes IBM Developed an instant messaging protocol , It may become an important part of the Internet of things . The protocol supports all platforms , It can connect almost all online items with the outside world , Used as sensors and actuators （ Such as through Twitter Let the house network ） Communication protocol of . 
advantage ： Simple format 、 Small bandwidth 、 Mobile communication 、PUSH、 Embedded system

      6.3 STOMP agreement

STOMP（Streaming Text Orientated Message Protocol） Is the streaming text oriented message protocol , It's for MOM(Message Oriented Middleware, Message Oriented Middleware ) Designed simple text protocol .STOMP Provide an interoperable connection format , Allow clients with any STOMP The message broker （Broker） Interact . 
advantage ： Command mode （ Not topic\queue Pattern ）

      6.4 XMPP agreement

XMPP（ Extensible message processing field protocol ,Extensible Messaging and Presence Protocol） It's based on extensible markup language （XML） The agreement , More for instant messaging （IM） And online live detection . For quasi instant operation between servers . The core is based on XML streaming , This protocol may eventually allow Internet users to send instant messages to anyone else on the Internet , Even if its operating system and browser are different . 
advantage ： General public 、 Strong compatibility 、 Scalable 、 High safety , but XML The coding format takes up a large amount of bandwidth

      6.5 The other is based on TCP/IP Custom protocol

Some special frames （ Such as ：redis、kafka、zeroMq etc. ） Not strictly following according to one's own needs MQ standard , It's based on TCP\IP Self encapsulating a set of agreements , Through the network socket Interface for transmission , Realized MQ The function of .

 

7 Common message middleware MQ Introduce

      7.1 RocketMQ

Alibaba is an open source distributed 、 Message middleware of queue model , Original name Metaq,3.0 The version name is changed to RocketMQ, It's Ali's reference kafka Use of design ideas java A set of implementations mq. At the same time, the Department of Ali will have a number of internal mq product （Notify、metaq） Integration , Maintain only core functions , Remove all other runtime dependencies , Make sure the core functions are the simplest , On this basis, we will cooperate with other open-source products mentioned above to realize different scenarios mq The architecture of , At present, it is mainly used in order trading system .

It has the following characteristics ：

• Can guarantee strict message order
• Provides filtering for messages
• Provides rich message pull patterns
• Efficient subscriber level scalability
• Real time message subscription mechanism
• 100 million message stacking capacity

The official offer is different from kafka The contrast difference ： 
https://rocketmq.apache.org/docs/motivation/

      7.2 RabbitMQ

Use Erlang Write an open source message queue , It supports a lot of agreements ：AMQP,XMPP, SMTP,STOMP, Exactly so , It's very heavyweight , More suitable for enterprise level development . At the same time Broker framework , The core idea is that producers don't send messages directly to queues , Messages are queued in the central queue when they are sent to the client . Routing (Routing), Load balancing (Load balance)、 Data persistence is well supported . More for enterprise level ESB Integrate .

      7.3 ActiveMQ

Apache Next sub project . Use Java Fully support JMS1.1 and J2EE 1.4 canonical JMS Provider Realization , A small amount of code can efficiently implement advanced application scenarios . Pluggable transport protocol support , such as ：in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports.RabbitMQ、ZeroMQ、ActiveMQ All support common multi language clients C++、Java、.Net,、Python、 Php、 Ruby etc. .

      7.4 Redis

Use C Language development Key-Value Of NoSQL database , Development and maintenance are very active , Although it is a Key-Value Database storage system , But it supports MQ function , So it can be used as a lightweight queue service . about RabbitMQ and Redis The entry and exit operations of , Each executive 100 Ten thousand times , Every time 10 Ten thousand times record the execution time . The test data is divided into 128Bytes、512Bytes、1K and 10K Four different sizes of data . Experiments show that ： When you join the team , When the data is small Redis The performance is higher than RabbitMQ, And if the data size exceeds 10K,Redis It's too slow to bear ; When the team , No matter the size of the data ,Redis All showed very good performance , and RabbitMQ The out of team performance is much lower than Redis.

      7.5 Kafka

Apache Next sub project , Use scala Implementation of a high-performance distributed Publish/Subscribe Message queuing system , It has the following characteristics ：

• Fast persistence ： Through the disk sequential read-write and zero copy mechanism , Can be in O(1) Message persistence under the system overhead of ;
• High throughput ： It can be achieved on a common server 10W/s The throughput rate of ;
• High accumulation ： Support topic The next consumer is offline for a long time , There's a lot of news piling up ;
• A completely distributed system ：Broker、Producer、Consumer All native automatic support distributed , rely on zookeeper Automatic realization of complex equilibrium ;
• Support Hadoop Data is loaded in parallel ： For things like Hadoop The same log data and offline analysis system , But it also requires the limitation of real-time processing , This is a viable solution .

      7.6 ZeroMQ

The fastest message queuing system , Dedicated to high throughput / Low latency scenario development , It is often used in financial applications , Focus on real-time data communication scenarios .ZMQ Can achieve RabbitMQ Not good at advanced / Complex queues , But developers need to assemble multiple technical frameworks themselves , High development cost . therefore ZeroMQ With a unique non middleware pattern , More like a socket library, You don't need to install and run a message server or middleware , Because your app itself uses ZeroMQ API Complete the role of logical Services . however ZeroMQ Only non persistent queues , If down machine , Data will be lost . Such as ：Twitter Of Storm Use in ZeroMQ As the transmission of data stream .

ZeroMQ Sockets are not related to the transport layer ：ZeroMQ Socket defines a unified... For all transport layer protocols API Interface . The default support In process (inproc) , Interprocess (IPC) , multicast ,TCP agreement , Switching between different protocols simply changes the prefix of the connection string . You can switch from local communication between processes to distributed communication at least at any time TCP signal communication .ZeroMQ In the back processing connection establishment , Disconnect and reconnect logic .

characteristic ：

• Lockless queue model ： For cross thread interactions （ The client and session） Data exchange channel between pipe, The queue algorithm without lock is adopted CAS; stay pipe There are asynchronous events registered at both ends of , Reading or writing messages to pipe When the , Will automatically trigger read and write events .
• Algorithms for batch processing ： For bulk messages , We optimized the adaptability , Can receive and send messages in batches .
• Thread binding under multi-core , need not CPU Switch ： Different from the traditional multithreading mode , Semaphore or critical area ,zeroMQ Make full use of the advantages of multi-core , Each core binding runs a worker thread , Avoid multithreading between CPU Switching overhead .

 

Two 、 Comparison of main message middleware

 

Comprehensive selection RabbitMq