Technology inventory: past, present and future of Message Oriented Middleware

The authors introduce ： Lin Qingshan （ flower ： Lonkie ）

operating system 、 database 、 Middleware is the troika of basic software , Message queue is one of the most classic middleware , There has been a 30 Years of history . Its development has mainly gone through the following stages ：

First stage ,2000 Years ago . 80 The first message queue was born in the s The Information Bus, Publish subscribe mode is proposed for the first time to solve the communication problem between software ; here we are 90 years , Is the era of international business software giants ,IBM、Oracle、Microsoft Have launched their own MQ, The most representative of them is IBM MQ, Expensive , For high-end enterprises , Such as large-scale finance 、 Telecommunications and other enterprises ; This kind of business MQ High end hardware is generally used , Software and hardware all-in-one machine delivery ,MQ The stand-alone architecture itself is .

The second stage ,2000~2007 year . Get into 00 s , The rise of the first generation of open source message queues , The birth of JMS、AMQP Two standards , The corresponding two implementations are ActiveMQ、RabbitMQ, Open source has greatly promoted the popularity of message queues , Lower the threshold of use , It has gradually become the standard configuration of enterprise architecture . Compared with today , This kind of MQ Mainly for traditional enterprise applications , For small traffic scenarios , The horizontal expansion ability is relatively weak .

The third stage ,2007~2018 year . PC Internet 、 The explosive development of mobile Internet . Because the traditional message queue can not bear the access traffic and massive data transmission of 100 million users , The birth of Internet messaging middleware , The core capability is to fully adopt distributed architecture 、 Have strong horizontal expansion ability , Typical representatives of open source include Kafka、RocketMQ, And Taobao Notify.Kafka The birth of has also transformed message oriented middleware from Messaging The field extends to Streaming field , From the asynchronous decoupling scenario of distributed applications to the streaming storage and streaming computing scenario in the field of big data .

The fourth stage ,2014~ so far . IoT、 Cloud computing 、 Cloud native leads new technology trends . oriented IoT Scene , The message queue starts communication from the cloud server application , Extend to edge computer rooms and Internet of things terminal equipment , Support MQTT Standard protocols such as Internet of things have also become the standard configuration of major message queues .

With the popularity of Cloud Computing , The concept of cloud native is deeply rooted in the hearts of the people , Various cloud native representative technologies emerge in endlessly , Include container 、 Microservices 、Serverless、Service Mesh、 Event driven, etc . The core problem of cloud native is how to redesign applications , In order to fully release the technological dividend of Cloud Computing , The shortest path to business success .

The message queue itself serves as the of cloud computing PaaS One of the services , We should give further play to “ decoupling ” The ability of , Help businesses build modern applications , The most critical capability evolution here is Eventing Evolution of . By sublimating the message into “ event ”, Provide standards oriented CloudEvent Orchestration filtering 、 Capabilities such as publish and subscribe build a wider range of decoupling functions , Including the decoupling of cloud service events and business applications 、 Cross organization SaaS Decoupling of business events 、 Decoupling of legacy applications and modern applications , At the same time, event driven is also a natural fit for cloud computing Serverless The paradigm of function calculation , It's an app Serverless Catalyst for chemical evolution .

Cloud native messaging middleware , Another meaning is the cloud evolution of the message queue architecture , How to give full play to cloud elastic Computing 、 Storage 、 The Internet , Let yourself get stronger technical indicators and Serverless Resilience .

What are the technological advances and breakthroughs of message oriented middleware ？

Alibaba cloud MQ Is based on RocketMQ Create a one-stop messaging service , With RocketMQ As a unified kernel , Achieve industry standards 、 Mainstream messaging protocols , Include MQTT、Kafka、RabbitMQ、AMQP、CloudEvent、HTTP etc. , Meet the demands of customers in diversified scenarios . To improve ease of use , We have made different protocols into products , Provide message service in the mode of independent product （ Such as ali cloud RabbitMQ、 Alibaba cloud Kafka）, Open the box 、 No operation and maintenance 、 Complete observable system , Help open source customers move seamlessly to the cloud .

After experiencing the continuous polishing of the diversified scenes of tens of thousands of enterprise customers , Years of production practice of super large-scale Cloud Computing , Our kernel RocketMQ Gradually evolve to integrated architecture and cloud native architecture .

1. Integrated architecture

Microservices 、 big data 、 Real time computing 、IoT、 Event driven technology trends , Constantly expand the business boundary of messages , There are different message queues in the industry to meet different business scenarios , such as RabbitMQ Focus on meeting the micro service scenario ,Kafka It focuses on meeting the needs of big data 、 Event flow scenario ,EMQ Is satisfied with IoT Vertical field scene . With the deepening of digital transformation , Customers' businesses often involve cross scenarios at the same time , For example, news from Internet of things devices 、 Or the business messages generated by the micro service system should be calculated in real time , If you are introducing multiple systems , Will bring additional machines 、 Operation and maintenance 、 Learning and other costs .

in the past “ branch ” It is often a compromise of technical realization , And now “ close ” The real needs of users are . RocketMQ 5.0 Based on Unification Commitlog Extended diversity index , Including time index 、 Million queue index 、 Transaction index 、KV Indexes 、 Batch index 、 Logical queue and other technologies . At the same time, it supports RabbitMQ、Kafka、MQTT、 Edge lightweight computing and other product capabilities , It's true “ news 、 event 、 flow ”,“ Cloud edge end ” Integrated architecture .

2.  Cloud native architecture

Cloud native architecture refers to the native architecture on the cloud , Cloud computing is cloud native “ Source power ”, Without cloud computing, talking about cloud nativity is like talking on paper .RocketMQ Over the past few years, it has been based on Alibaba cloud's super large-scale cloud computing production practice , Learn from the experience of helping tens of thousands of enterprises complete digital transformation , So as to complete the evolution from Internet messaging middleware to cloud native messaging middleware . This is also RocketMQ The biggest difference from other message oriented middleware , It is the cloud native architecture that has been put into practice , Let's take stock of RocketMQ Key technology evolution in cloud native architecture .

RocketMQ yes 2011 Born in the core e-commerce system of Taobao , At the beginning, it was positioned to serve the business of the group , Design for a single super large-scale Internet enterprise . The original architecture can not well meet the cloud computing scenario , There are many pain points , For example, heavy SDK, The client logic is complex 、 Multilingual SDK High development cost 、 Business feature iteration is slow ; Poor flexibility , Compute storage coupling 、 The number of clients and physical queues is coupled 、 The number of queues cannot be expanded to millions 、 Tens of millions ; Other mainstream open source news projects have also not carried out the transformation of cloud native architecture , such as RabbitMQ Single queue capacity cannot scale out 、Kafka Elastic capacity expansion will face a large number of data copy balancing , Are not applicable to providing flexible services for large-scale customers in the public cloud .

So ,RocketMQ 5.0 Redesign the scene for Cloud Computing , Expect to solve the fundamental problems from the architecture level , Client side 、Broker To a comprehensive upgrade of the storage engine ：

Client lightweight . RocketMQ 5.0 SDK Sink a lot of logic to the server , Reduce the number of lines of code by two-thirds , Develop and maintain multilingualism SDK Greatly reduce the cost of ; A lightweight SDK It's easier to be Service Mesh、Dapr Such as cloud native technology integration .

Separable and separable storage and computing separation architecture . Users appeal according to different scenarios , You can start the functions of storage and calculation in the same process , You can also deploy the two separately . The separately deployed computing nodes can achieve “ No state ”, One access point can proxy all traffic , Combine new hardware kernel bypass technology on the cloud , It can reduce the performance and delay problems caused by separate deployment . And choose “ Storage and computing integration ” framework , It has “ Calculate nearby ” The advantages of , Better performance . Rent more on the cloud 、 many VPC Complex networks 、 In the scenario of multi protocol access , The separation of storage and computing mode can avoid the direct exposure of back-end storage services to clients , Facilitate flow control 、 Isolation 、 Dispatch 、 Rights management 、 Protocol conversion, etc. .

But there are advantages and disadvantages , The separation of storage and computing also leads to the lengthening of the link 、 Delay increases 、 Problems such as rising machine costs , Operation and maintenance has not been simplified , In addition to maintaining stateful storage nodes , More operation and maintenance stateless computing nodes . In fact, in most simple messaging scenarios , The data link is basically written Log、 read Log, No complex computational logic （ The calculation logic is too simple compared with the database ）, At this time Integrated storage and computing architecture , Simple enough 、 High performance 、 Low latency . Especially in the big data transmission scenario , The integration of storage and computing can greatly reduce the cost of machine and flow , This from Kafka The evolution of architecture can also be confirmed . In general, don't separate for the sake of saving and calculating , Or return to customers 、 The essence of the business scenario .

Elastic storage engine . oriented IoT Mass devices 、 Large scale small customer scenarios on the cloud , We introduce LSM Of KV Indexes , Realize the ability of single machine mass queue , The number of queues can expand indefinitely ; In order to further release the capacity of cloud storage , We implement tiered storage , The message storage time is from 3 Days rise to months 、 Grade , The storage space can be expanded infinitely , At the same time, the cold and hot data are separated , The cost of cold data storage is reduced 80%.

Serverless turn . In the old architecture , Customer aware physical queues , The physical queue is bound to a fixed storage node , Strong state .Broker、 client 、 The expansion and contraction capacities of physical queues are coupled with each other , The granularity of load balancing is queue level , Yes Serverless The evolution of technology is very unfriendly . For maximum flexibility Serverless,RocketMQ 5.0 Further decouple logical resources and physical resources .

stay Messaging/ The scenario of unordered messages , Customer designation Topic Send and receive messages out of order , The new architecture shields the concept of queues from clients , Expose only logical resources Topic. The granularity of load balancing ranges from queue level to message level , It realizes the stateless of the client , client 、 The elastic expansion of the server is decoupled .

stay Streaming/ Scenario of sequential messages , The client needs to specify Topic A queue under （ Also called zoning ） Send and receive messages in sequence . In the new architecture , Mask physical queues for clients , Introduce the concept of logical queue , A logical queue is segmented horizontally and vertically , Physically dispersed nodes . Horizontal slicing solves the problem of high availability , Multiple shards and multipoints of the same logical queue are randomly writable , be based on Happen before The principle of preserving order , Second level Failover, No active / standby switching is required ; Longitudinal segmentation , Solve the problem of logical queue expansion , Through multi-level queue mapping , Realization 0 Second scale expansion of data migration , Elastic scaling decoupling of logical resources and physical resources .

How to view the ecological players in the news field ?

In cloud native 、IoT、 Guided by the trend of big data , Message has become the rigid demand of modern application architecture , More extensive use scenarios , It can be applied to asynchronous decoupling of microservices 、 Event driven 、 Internet of things device data uplink and downlink 、 Big data stream storage 、 Scenarios such as lightweight flow computing . Strong customer demand 、 Active market , Attracted many manufacturers to join the competition .

From a good point of view , Full competition of manufacturers , Will further activate innovation , Train more users , Make a big news market together , Users also seem to have more choices ;

From a bad point of view , In the future, some message queues that fail in competition will enter a stagnation period 、 Offline period , Users' applications will face the risk of migration, transformation and stability , Therefore, it is recommended that users meet their own business needs , Try to choose standard interfaces 、 Protocol access , Or directly use the industry fact standard message queue .

What is the future development trend of message oriented middleware ？

With IoT、5G The sustainable development of the Internet , Data volume growth 28%, Expect to 2025 Internet of things devices will reach 400 One hundred million units , Enter the era of interconnection of all things . In the era of Internet of things, the amount of message storage and computing will grow explosively , The message system will face huge cost pressure . Future messaging systems , need Dig deep into the dividends of new hardware , Such as persistent memory 、DPU Technology , The combination of software and hardware is adopted to deeply optimize , Further reduce the storage and computing cost of messages .

IoT Another important trend of the times is Edge of computing ,Gartner Expect to 2025 year ,75% The data will be processed outside the traditional data center or cloud environment , The messaging system needs to be further lightweight 、 Reduce resource consumption to adapt to the edge computing environment . It also means that , Integrated architecture of message oriented middleware , Have good plug-in design , Multi modality output can be realized according to the characteristics of the scene . For example, the form of public cloud can be deeply integrated with the infrastructure of public cloud , Make full use of cloud disk 、 Object storage enhances storage capacity , Integrated log service 、 Application monitoring and other services to improve observability ; The form of edge computing is to output core storage with the least resource cost 、 The power of lightweight Computing , Simple enough .

In recent years, cloud computing has developed rapidly , Thanks to the digital transformation of a large number of enterprises around the world , Through business online 、 Business data 、 Data intelligence to enhance the competitiveness of enterprises . Data transformation is also accompanied by the transformation of business thinking , More and more enterprises adopt “ Event driven ” To build business logic and digital systems .

Gartner forecast , The future surpasses 60% New digital business solutions will adopt “ Event driven ” Pattern , From a business perspective ,“ Event driven ” The model can help enterprises respond to customers in real time , Seize more business opportunities , Create incremental value ; From a technical point of view ,“ Event driven ” The architecture of , Be able to dynamically 、 flexible 、 A decoupled way to link across organizations 、 Heterogeneous systems across environments , Naturally, it is suitable for building a large-scale cross organizational digital business ecosystem .

In response to this trend ,Messaing Go to Eventing evolution , There is EventBridge （EventBroker） Product form . stay EventBridge in ,“ event ” The concept of becoming a first-class citizen , Event publishers and subscribers are not coupled to any specific message queue SDK And the implementation .EventBroker Around the standard CloudEvent Standardize the construction of a more generalized publish subscribe model , Be able to link everything across organizations 、 Heterogeneous event sources and event processing targets across environments .

At present “ Event driven ” The construction of digital business ecology has just begun , future EventBridge More powerful capabilities will be implemented around the level of abstraction of events , For example, the whole link of events can be observed 、 Event analysis and calculation 、 Low code development and other features , Help enterprises fully land in the cloud era “ Event driven ” framework .

The authors introduce ：

​ Lin Qingshan （ flower ： Lonkie ）, Alibaba cloud senior technical expert , Alibaba cloud message product line leader . International information experts , Committed to the message 、 Real time computing 、 Research and exploration in event driven and other directions , advance RocketMQ Cloud native architecture 、 Evolution of hyper fusion architecture .​