How emqx 5.0 under the new architecture of mria+rlog realizes 100million mqtt connections

introduction ： Single cluster 1 Billion MQTT Connection reached

not long ago , Open source large-scale distributed Internet of things MQTT Message server

EMQX Released 5.0 edition

.

This latest milestone version uses a new back-end storage architecture Mria database , And reconstruct the data replication logic , therefore

EMQX 5.0 The horizontal expansion ability has been improved exponentially , It can carry more large-scale Internet of things device connections more reliably .

stay EMQX5.0 In the performance test before the official release , We passed a 23 Node EMQX colony , For the first time in the world 1 Billion MQTT Connect + Per second 100 Million message throughput , It also makes

EMQX 5.0 So far, it has become the most prominent and expansible MQTT Broker.

This article will EMQX The new underlying architecture with exponential improvement in horizontal expansion ability is analyzed in detail , Help you understand EMQX 5.0 The technical principle of cluster expansion , And how to choose the appropriate deployment architecture in different practical application scenarios , Realize more reliable device access and message transmission .

4.x Time ： Use Mnesia structure EMQX colony

Mnesia Introduce

EMQX 4.x Version storage adopts Erlang/OTP Built in distributed database Mnesia（https://www.erlang.org/doc/man/mnesia.html）, It has the following advantages ：

• Embedded：
  and MySQL、PostgeresSQL When the database is different ,Mnesia and EMQX Is running in the same operating system process （ Be similar to SQLite）. therefore EMQX You can read routes very fast 、 Conversation and other related information .

• Transactional：
  Mnesia Support transactions and have ACID Guarantee . And these guarantees are effective for all nodes of the entire cluster .EMQX Use where data consistency is important Mnesia Business , For example, update the routing table 、 Create rules engine rules, etc .

• Distributed：
  Mnesia The table will be copied to all EMQX node . This can improve EMQX  Distributed fault tolerance , As long as a node survives, the data is safe .

• NoSQL：
  Traditional relational databases use SQL Interact with the database . and Mnesia Use it directly Erlang Read and write expressions and built-in data types , This makes the integration with business logic very smooth , And eliminate the overhead of data encoding and decoding .

stay Mnesia In the cluster , All nodes are equal . Each of them can store a copy of data , You can also start a transaction or perform read-write operations .

Mnesia The cluster uses a full mesh topology ： That is, each node will establish connections with all other nodes in the cluster , Each transaction is replicated to all nodes in the cluster . As shown in the figure below ：

Mnesia The problem of

As we discussed above ,Mnesia Databases have many very significant advantages ,EMQX It also gains a lot from it . But its fully connected characteristics ,

It limits the horizontal expansion ability of its cluster , Because the number of links between nodes increases with the square of the number of nodes , The cost of keeping all nodes fully synchronized is getting higher and higher , The performance of transaction execution will also decline dramatically .

It means EMQX The clustering function of has the following limitations :

• Insufficient horizontal expansion capacity .
  stay 4.x We do not recommend too many nodes in the cluster , Because the overhead of transaction replication in the mesh topology will be larger and larger ; Our general recommendation is to keep the number of nodes at 3~7 individual , And try to provide single node performance .

• An increase in the number of nodes will increase the possibility of cluster brain fissure .
  The more nodes 、 The number of links between nodes will also increase dramatically , Higher requirements for network stability between nodes . When there is a brain fissure , Node self-healing will lead to node restart and risk of data loss .

For all that ,EMQX With its unique architecture design and Erlang/OTP Powerful features , Realize a single cluster 1000 ten thousand MQTT The target of the connection . meanwhile ,EMQX It can be bridged by clusters , Carry larger scale IOT applications through multiple clusters .

but

With the development of the market , A single Internet of things application needs to carry more and more devices and users

,EMQX It needs to have stronger scalability and access capability , To support super large-scale Internet of things applications .

5.x Time ： Use Mria Building large-scale clusters

Mria（https://github.com/emqx/mria） yes Mnesia An open source extension of , Adds final consistency to the cluster . Most of the features described above still apply to it , The difference lies in how data is replicated between nodes . 

Mria from

Full mesh

Topology turns

Reticular + Star shaped

topology . Each node assumes one of two roles ：

Core nodes （Core） Or replicator node （Replicant）.

Core and Replicant Node behavior

Core Node behavior and 4.x Medium Mnesia The nodes are the same ：Core Nodes form a cluster in a fully connected way , Each node can initiate transactions 、 Hold a lock, etc . therefore ,EMQX 5.0 Still require Core Nodes should be as reliable as possible in deployment .

Replicant Nodes are no longer directly involved in transaction processing . But they connect to Core node , And passively copy from Core Data update of node .Replicant Nodes are not allowed to perform any write operations . Instead, it is handed over to Core The node executes on behalf of .

in addition , because Replicant Will copy from Core Node data , So they have a complete copy of the local data , To achieve the highest efficiency of read operations , such

Help to reduce EMQX Routing delay .

We can treat this data replication model as a mixture of unowned replication and master-slave replication . This cluster topology solves two problems ：

• Horizontal scalability （ As mentioned earlier , We have tested there 23 Of nodes EMQX colony ）

• Easier automatic cluster expansion , There is no risk of data loss

because Replicant Nodes do not participate in write operations , When more Replicant When a node joins the cluster , The latency of write operations will not be affected . This allows you to create larger EMQX colony .

in addition ,Replicant Nodes are designed to be stateless . Adding or deleting them will not cause the loss of cluster data 、 It will not affect the service status of other nodes , therefore Replicant Nodes can be placed in an auto expansion group , thus

Achieve better DevOps practice .

For performance reasons , The replication of irrelevant data can be divided into independent data streams , That is, multiple related data tables can be assigned to the same RLOG Shard（ Copy log fragment ）, Sequence transactions from Core Nodes are copied to Replicant node . But different RLOG Shard It's asynchronous between .

EMQX 5.0 Cluster deployment practice

Cluster architecture selection

 

stay EMQX 5.0 in , So if you don't make any adjustments, all nodes default to Core node , Default behavior and 4.x The versions are consistent .

Can be set by emqx.conf Medium node.db_role Parameter or EMQX_NODE__DB_ROLE environment variable , Set the node to Replicant node .

*  Please note that , There must be at least one core node in the cluster , We suggest that 3 individual Core+N individual Replicant As a start

Core Node acceptable MQTT Business flow of , It can also be used purely as a cluster database . We suggest ：

• In a small cluster
  （3 Nodes or less ）, There is no need to use Core+Replicant Copying model , It can make Core Nodes undertake all traffic , Avoid increasing the difficulty of getting started and using .

• In a super large cluster
  （10 Nodes or more ）, Make a proposal to MQTT Flow from Core Remove the node , This is more stable and has better horizontal scalability .

• In a medium-sized cluster
  , Depends on many factors , You need to test according to the actual scenario of users to know which is better .

exception handling

Core Nodes for Replicant Nodes are insensitive , When a certain Core When the node goes down ,Replicant The node will automatically connect to the new Core node , During this process, the client will not be disconnected , But it may cause delay in route update ; When Replicant When the node goes down , All clients connected to this node will be disconnected , But because of Replicant It's stateless , So it will not affect the stability of other nodes , At this time, the client needs to set the reconnection mechanism , Connect to another Replicant node .

Hardware configuration requirements

The Internet

Core Network delay between nodes 10ms following , The measured value is higher than 100ms Will not be available , Please put Core Nodes are deployed in the same private network ;Replicant And Core It is also recommended that nodes be deployed under the same private network , But the network quality requirements can be compared with  Core Slightly lower between nodes .

CPU With memory

Core Nodes require large memory , Without connection ,CPU Low consumption ;Replicant Node hardware configuration and 4.x Agreement , Its memory requirements can be estimated by connection and throughput configuration .

Monitoring and commissioning

Yes Mria Performance monitoring of can be used Prometheus Or use EMQX Console view .Replicant The node will experience the following states during startup ：

• bootstrap：
  When Replicant After the node is started , Need from Core The process of node synchronizing the latest data table

• local_replay：
  When the node is finished bootstrap when , It must replay the write transactions generated in this process

• normal：
  When the cached transaction is fully executed , The node enters the normal operation state . Subsequent write transactions are applied to the current node in real time . Most of the time ,Replicant Nodes will remain in this state .

Prometheus monitor

Core node

• emqx_mria_last_intercepted_trans： Since the node was started , The number of transactions received by sub regions . Please note that , This value may be different on different core nodes .

• emqx_mria_weight： A value for load balancing . Its change depends on the instantaneous load of the core node .

• emqx_mria_replicants： The number of replicators connected to the core node , Copy data for a given slice .

• emqx_mria_server_mql： Number of unprocessed transactions , Waiting to be sent to the replicator . The less, the better. . If this indicator has an increasing trend , Need more core nodes

Replicant node

• emqx_mria_lag： Replicator lag , Indicates the extent to which the replicator lags behind the upstream core node . The less, the better. .

• emqx_mria_bootstrap_time： The time spent in the process of starting the replica . This value will not change during the normal operation of the replica .

• emqx_mria_bootstrap_num_keys： The number of database records copied from the core node during boot . This value will not change during the normal operation of the replica .

• emqx_mria_message_queue_len： The message queue length of the replication process . It should always be 0 about .

• emqx_mria_replayq_len： The length of the internal replay queue of the replica . The less, the better. .

Console command

./bin/emqxevalmria_rlog:status(). You can get information about Mria More information about the running status of the database .

* notes ： It can show some shard by down state , This indicates that these fragments are not used by any business applications .

Conclusion

The new underlying architecture makes EMQX 5.0 It has stronger horizontal expansion ability , While building larger clusters that meet business needs , It can reduce the risk of cerebral fissure under large-scale deployment and the impact after cerebral fissure , Effectively reduce cluster maintenance overhead , Provide more stable and reliable Internet of things data access services .