Common message oriented middleware selection

What are the mainstream Message Oriented Middleware ？

At present, the popular open source message middleware in the industry includes ：ActiveMQ、RabbitMQ、RocketMQ、Kafka、ZeroMQ etc. .

Among them, the most widely used is RabbitMQ、RocketMQ、Kafka These three paragraphs .

Redis To some extent, it can also achieve similar “Queue” and “ Pub/Sub” The mechanism of , Strictly speaking, it is not message oriented middleware .

General business systems to be introduced MQ, At first, everyone used ActiveMQ, But people don't really use it much anymore , The large throughput scenario is not validated , The community is not very active either , So let's forget it , This is not recommended .

Later, people began to use RabbitMQ, But it is erlang Language prevents a lot of Java Engineers to study it and master it , For the company , Almost out of control , But they are open source , More stable support , It's also very active .

But there are more and more companies , Will go to use RocketMQ, It's really nice （ Ali ' ）, But the community may be at risk of a sudden meltdown , Have absolute confidence in oneself company technical strength , Recommend to use RocketMQ, Otherwise go back to old honest practical RabbitMQ Well , They have an active open source community , Absolutely not yellow .

So small and medium-sized companies , The technical strength is relatively average , The technical challenges are not particularly high , use RabbitMQ It's a good choice ; Big companies , Infrastructure research and development strength is strong , use RocketMQ It's a good choice .

If it is the big data field of real-time computing 、 Log collection and other scenarios , use Kafka It's industry standard , Absolutely no problem , The community is very active , Absolutely not yellow , Moreover, it is the DE facto norm in this field almost all over the world .

One 、ActiveMQ

Apache Next sub project . Use Java Fully support JMS1.1 and J2EE 1.4 canonical JMS Provider Realization , A small amount of code can efficiently implement advanced application scenarios . Pluggable transport protocol support , such as ：in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports.

ActiveMQ、RabbitMQ、ZeroMQ All support common multi language clients C++、Java、.Net,、Python、 Php、 Ruby etc. .

ActiveMQ Only operating system support is required Java virtual machine , It can be carried out .

ActiveMQ Can be easily embedded to use Spring The system passed the common J2EE Server testing .

JMS namely Java Message service （Java Message Service） Application program interface , It's a Java About message-oriented middleware in the platform （MOM） Of API, Used between two applications , Or send a message in a distributed system , Asynchronous communication . It's rich in API 、 A variety of cluster construction modes make it a well-known message middleware in the industry , It is widely used in small and medium-sized enterprises ！

advantage ：

• ActiveMQ Using message push mode , So the most suitable scenario is that the default messages can be consumed in a short time . More data , The slower it is to find and consume messages , The degree of message backlog is inversely proportional to the speed of the message .

shortcoming ：

• Low throughput . because ActiveMQ Index required , Resulting in throughput degradation . This is an insurmountable shortcoming , As long as you use it exactly in accordance with JMS Standard message middleware , You have to accept this level of TPS.
• No slicing function . It's a loss of function ,JMS There is no provision for the clustering of message middleware 、 Fragmentation mechanism . And because the ActiveMQ It is a message middleware developed and designed by Wei enterprise , The original intention is not to handle massive messages and high concurrency requests . If a server can't take more messages , You need to split horizontally .ActiveMQ The government does not provide a fragmentation mechanism , You have to implement it yourself .

Two 、RabbitMQ

Use Erlang Write an open source message queue , It supports a lot of agreements ：AMQP,XMPP,SMTP,STOMP, Exactly so , It's very heavyweight , More suitable for enterprise level development . At the same time Broker framework , The core idea is that producers don't send messages directly to queues , Messages are queued in the central queue when they are sent to the client . Routing (Routing), Load balancing (Load balance)、 Data persistence is well supported . More for enterprise level ESB Integrate .

RabbitMQ It was originally used for reliable communication of telecommunication services , It is also a rare support AMQP One of the products of the agreement .

advantage ：

1. Lightweight , Fast , Easy to deploy and use
2. Support flexible routing configuration .RabbitMQ in , There is a switch module between the producer and the queue . According to the configured routing rules , Messages sent by producers can be sent to different queues . Routing rules are flexible , You can also realize .
3. RabbitMQ The client supports most programming languages .

shortcoming ：

1. If there are a lot of messages piled up in the queue , Performance can degrade dramatically
2. RabbitMQ Performance in Kafka and RocketMQ Is the worst , Processing tens of thousands to hundreds of thousands of messages per second . If the application requires high performance , Do not choose RabbitMQ.
3. RabbitMQ yes Erlang Developed , The cost of function expansion and secondary development is very high .

3、 ... and 、RocketMQ

Alibaba is an open source distributed 、 Message middleware of queue model , Original name Metaq,3.0 The version name is changed to RocketMQ, It's Ali's reference kafka Use of design ideas java A set of implementations mq. At the same time, the Department of Ali will have a number of internal mq product （Notify、metaq） Integration , Maintain only core functions , Remove all other runtime dependencies , Make sure the core functions are the simplest , On this basis, we will cooperate with other open-source products mentioned above to realize different scenarios mq The architecture of , At present, it is mainly used in order trading system .

It has the following characteristics ：

• Can guarantee strict message order
• Provides filtering for messages
• Provides rich message pull patterns
• Efficient subscriber level scalability
• Real time message subscription mechanism
• 100 million message stacking capacity

The official offer is different from kafka The contrast difference ：
https://rocketmq.apache.org/docs/motivation/

advantage ：

1. RocketMQ It is mainly used for ordering , Business , Flow calculation , Message push , Log stream processing ,binlog Distribution and other scenarios . After all previous double 11 test , performance , Stability is not reliable .
2. RocketMQ It has almost all the features and functions that message queuing should have .
3. java Development , Read source code 、 Expand 、 Secondary development is very convenient .
4. Many optimizations have been made for the response delay in the e-commerce field . in the majority of cases , The response is in milliseconds . If the application is concerned about response time , have access to RocketMQ.
5. Performance ratio RabbitMQ One order of magnitude higher , Processing hundreds of thousands of messages per second .

shortcoming ：

1. The integration and compatibility with peripheral systems are not very good .

Four 、Kafka

Apache Next sub project , Use scala and java Implementation of a high-performance distributed Pub/Sub Message queuing system , It has the following characteristics ：

• Fast persistence ： Through the disk sequential read-write and zero copy mechanism , Can be in O(1) Message persistence under the system overhead of ;
• High throughput ： It can be achieved on a common server 10W/s The throughput rate of ;
• High accumulation ： Support topic The next consumer is offline for a long time , There's a lot of news piling up ;
• A completely distributed system ：Broker、Producer、Consumer All native automatic support distributed , rely on zookeeper Automatic realization of complex equilibrium ;
• Support Hadoop Data is loaded in parallel ： For things like Hadoop The same log data and offline analysis system , But it also requires the limitation of real-time processing , This is a viable solution .

Kafka The reliability of the , The stability and functional characteristics basically meet most application scenarios . The compatibility with peripheral systems is one of the best , Especially in the field of big data and stream computing , Almost all relevant open source software supports Kafka.Kafka yes Scala and Java Developed , A lot of design has been done for batch processing and asynchronous processing , therefore Kafka You can get very high performance . Its asynchronous message sending and receiving is the best , But with RocketMQ Can't open the order of magnitude , Processing hundreds of thousands of messages per second . If it's an asynchronous message , And turn on compression ,Kafka Finally, it can be processed per second 2000w The level of the message .

advantage ：

1. Support multiple producers and consumers
2. Support broker The horizontal expansion of
3. Replica set mechanism , Achieve data redundancy , Ensure that data is not lost
4. adopt topic Classify data
5. By sending compressed data in batches , Reduce data transmission overhead , Increase swallowing volume
6. Messages that support multiple modes
7. Data Persistence Based on disk
8. High performance processing of information , In the case of big data , It can guarantee sub second message delay
9. One consumer can support multiple topic The news of
10. Yes CPU And memory consumption is relatively small
11. The network overhead is also relatively small
12. Support data replication across data centers
13. Support mirror cluster

shortcoming ：

1. Because it's bulk delivery , So the data is not really real-time
2. about mqtt Protocol not supported
3. It does not support the direct access of IOT sensor data
4. It can only support ordered messages in a unified partition , Unable to achieve global message ordering
5. Monitoring is not perfect , Need to install plug-ins
6. Need to cooperate with zookeeper Metadata management
7. Data will be lost , And transactions are not supported
8. There may be repeated consumption data , The news will be out of order , Can guarantee a fixed partition The internal news is orderly , But one topic There are many. partition Words , There's no guarantee of order , need zookeeper Support for ,topic Generally, you need to create , Deployment and maintenance are generally better than mq high

5、 ... and 、ZeroMQ

The fastest message queuing system , Dedicated to high throughput / Low latency scenario development , It is often used in financial applications , Focus on real-time data communication scenarios .ZMQ Can achieve RabbitMQ Not good at advanced / Complex queues , But developers need to assemble multiple technical frameworks themselves , High development cost . therefore ZeroMQ With a unique non middleware pattern , More like a socket library, You don't need to install and run a message server or middleware , Because your app itself uses ZeroMQ API Complete the role of logical Services . however ZeroMQ Only non persistent queues , If down machine , Data will be lost . Such as ：Twitter Of Storm Use in ZeroMQ As the transmission of data stream .

ZeroMQ Sockets are not related to the transport layer ：ZeroMQ Socket defines a unified... For all transport layer protocols API Interface . The default support In process (inproc) , Interprocess (IPC) , multicast ,TCP agreement , Switching between different protocols simply changes the prefix of the connection string . You can switch from local communication between processes to distributed communication at least at any time TCP signal communication .ZeroMQ In the back processing connection establishment , Disconnect and reconnect logic .

characteristic ：

• Lockless queue model ： For cross thread interactions （ The client and session） Data exchange channel between pipe, The queue algorithm without lock is adopted CAS; stay pipe There are asynchronous events registered at both ends of , Reading or writing messages to pipe When the , Will automatically trigger read and write events .
• Algorithms for batch processing ： For bulk messages , We optimized the adaptability , Can receive and send messages in batches .
• Thread binding under multi-core , need not CPU Switch ： Different from the traditional multithreading mode , Semaphore or critical area ,zeroMQ Make full use of the advantages of multi-core , Each core binding runs a worker thread , Avoid multithreading between CPU Switching overhead .

6、 ... and 、 Comparison of main message middleware