当前位置：网站首页>Design of Distributed Message Oriented Middleware

Single application ： All of the code 、 modular , All in one application . If one of the modules is to be upgraded , Then the whole system should be upgraded . If deployed in ten machines , A small function has been changed , All ten machines have to be upgraded .
Facing problems ： High coupling degree 、 Development is difficult , There are many problems

1.2、 Distributed system architecture

Distributed systems ： Put a large system according to Business module 、 Function module Technical division of , Divided into several independent Subsystem .

In the following figure, a large system is divided into a foreground system 、 Order system 、 Member system , The systems are independent of each other 、 Independent deployment . Upgrade a system , Other subsystems do not need to be adjusted .

The user sends a request to the foreground system , A user's request cannot be completed by a foreground system .
for example ： Users need their own order information and member information , Send the request to the foreground system , The front desk system needs the system order system and the member system , Process this request .
A scenario where multiple systems work together to process a request , We call it Distributed systems .

There are also calls between systems , This requires rpc（ Remote procedure call ） technology . The way to do it , There is strong coupling between the systems . To achieve decoupling , Implement a more extensible architecture , So in the distributed system Message middleware .

Solve the coupling between systems through message oriented middleware

2、 Distributed system architecture based on Message Oriented Middleware

The user sends a request to create an order , It is not handled directly by the order system .
User send create The order request is sent to the front desk system , The foreground system passes the request to Message middleware After staging , It is passed to the corresponding subsystem by the message oriented middleware for processing , Then perform a client-side corresponding .

2.1、 What is message middleware

Message middleware ： Temporarily store some messages 、 Some data , Independently deployed , So it is called middleware . Used between systems , For data exchange .

After the message oriented middleware is used between systems , There is no direct connection , It doesn't call , Transformation by message oriented middleware .
Data exchange between systems ： You can use the interface to directly call 、 Message oriented middleware can also be used for asynchronous processing , Asynchronous distribution .

2.2 Overview of message middleware

What is message middleware

① Make use of efficient and reliable messaging mechanism for platform independent data exchange .
② And the integration of distributed system based on data communication .
③ By providing message mechanism and message queuing model , It can expand communication between processes in a distributed environment ;

Application scenarios of message middleware

Cross system data transfer
Peak clipping of high concurrent traffic
Data asynchronous processing, etc

for example ： Now there is 1 Ten thousand pieces of data need to be inserted into the database , Direct insertion into the database is under great pressure , You can cache the inserted data in the message oriented middleware , Asynchronous processing , Traffic peak clipping , Reduce concurrency .

Common message middleware

ActiveMQ、RabbitMQ、Kafka、RocketMQ

These are all web-based , Based on message passing mechanism , That is, the scenario of producers and consumers .

3、 The core design of message oriented middleware

How do we write message oriented middleware ？

3.1、 The essence

One that can receive requests 、 Save the data 、 Network applications with functions such as sending data .
And general network applications ： It is mainly responsible for receiving and transmitting data , So the performance is generally higher than that of ordinary programs .

3.2、 Five core components

agreement
Persistence mechanism
Message distribution mechanism
High availability design
High reliability design

3.2.1、 agreement

agreement It is a set of conventions that computer communication follows together , All follow the same agreement , Computers can communicate with each other .
It is a formal description of the data format and the rules that must be followed when exchanging data between computers .

for example ： Server side xml data , The client with json Parsing data . It cannot be resolved without following the same protocol .

Three elements of agreement ：

grammar ： Structure and format of data and control information .（ What is the structure of data transmission ）
semantics ： What kind of control messages need to be sent out , What actions to complete and what responses to make （ What kind of message to send ）
sequential （ Sync ）： A detailed description of the order in which events are implemented

Common agreements

Http Three elements of agreement ：

grammar ： The specific format of request message and response message is specified .(nio、bio)
semantics ： The operation initiated by the client is called a request ;（post、get request ...）
sequential ： A request corresponds to a response (request、respond）

http Not suitable for message oriented middleware MQ To use ,

Message middleware commonly used protocol ：

OpenWire（ActiveMQ exclusive ）、AMQP、MQTT、Kafka、OpenMessage

Why doesn't message middleware use Http agreement ？

①http There are many requests , A simple request needs to pass a pile of data , The syntax format is complex , Contains the request header 、Coking Etc , Status code information, etc . Message middleware , What is needed is a single , Try to be as concise as possible 、 Try to reduce the size of the data we pass . This is why most message oriented middleware does not use Http Why ,Http It's too big .
②Http Most of the time Short connection . After each interaction request response , It will be interrupted , Scenarios that are not conducive to message oriented middleware . Because the message oriented middleware may be a client that acquires information in the message oriented middleware for a long time , Or send data to the message oriented middleware , Not suitable for short connections

AMQP agreement ：

Advanced Message Queuing Protocol Advanced message queue protocol .

04 year Jpmorgan Chase JPMorgan Chase and other companies jointly designed .

characteristic ：

Transaction support 、 Persistence support , Born in the financial industry , It has a natural advantage in reliable message processing .

MQTT agreement ：

Open Message agreement

Kafka agreement ：

3.2.2、 Persistence

Simply put, it's storing data on disk , Not in memory , Disappear with service restart , Making data permanent is called persistence .

No persistence , The server went down suddenly , The unprocessed messages in the message oriented middleware will disappear .

Common persistence methods

Write to log file , Store in a specific format . Or use the database scenario .

3.2.3 Message delivery

Data in message oriented middleware , Push the message to the consumer . The consumer pulls data from the message queue .

Why there is a message distribution strategy ？

A message oriented middleware links multiple systems , Which system should messages in message oriented middleware be sent to ？ According to what kind of strategy exactly send to the corresponding system

For the first time to the consumer , Handling errors , Can I resend , Continue distribution processing

Common message oriented middleware distribution strategies

3.2.4 High availability

High Availability Mechanism

High availability refers to the ability of the product to perform the specified functions under the specified conditions and within the specified time or time interval .
When there's a lot of business , A message oriented middleware may not meet the requirements , Therefore, message oriented middleware is required to be able to be deployed in clusters , To achieve high availability .

Master-Slave The deployment of master-slave shared data ：

Lord broker get data , Message store , other broker Sure share Read .

Master-Slave Master-slave Sync Deployment way ：

The producer sends data , Will store data in Master and slave servers . Three machines serve the outside world at the same time , Solve the load balancing problem of reading data . Producer data insert modifications can only be performed on the primary server .

Synchronization takes up a lot of bandwidth .

Broker-Cluster Multi master cluster synchronous deployment mode

Are all primary servers , It can synchronize data with each other

Whether it's reading or writing , Can load balance

Broker-Cluster Multi master cluster forwarding deployment mode

Different metadata is stored in different broker On , Metadata description information is synchronized in each broker On .

Request pull broker-1 When a piece of data on ,broker-1 There is no such data on , But it knows broker-2 There is this data on the , So it was forwarded to broker-2 On .