Message queue: how to deal with message backlog?

Catalog

One 、 Preface

Two 、 Optimize performance to avoid message backlog

1. Send end performance optimization

2. Consumer performance optimization

3、 ... and 、 How to deal with the backlog of messages ?

Four 、 Summary

---

One 、 Preface

In the problem of using message queuing , The problem of message backlog , It should be the most common problem , also , This problem It's not easy to solve .

We all know , The direct cause of the message backlog , There must be a performance problem in some part of the system , There's no time to deal with the upstream transmission Rest , That leads to a backlog of messages .

therefore , Let's analyze it first , When using message queuing , How to optimize code performance , Avoid message backlog . Then we'll see , If your online system has a message backlog , How to deal with the emergency , Minimize the impact of message backlog on the business .

Two 、 Optimize performance to avoid message backlog

In systems that use message queuing , For performance optimization , It is mainly reflected in the business logic of the producer and the consumer . For the performance of the message queue itself , You as a user , Don't pay too much attention to . Why do you say that ?

The main reason is , For the vast majority of services that use message queuing , The processing power of message queue itself is far greater than that of business system force . Single node of the mainstream message queue , The performance of message sending and receiving can reach the level of processing tens of thousands to hundreds of thousands of messages per second , It's OK Over horizontal expansion Broker The number of instances multiplied the processing power .

But the business logic that the general business system needs to deal with is far more complex than the message queue , A single node can handle hundreds to thousands of requests per second , It's already very good performance . therefore , For performance optimization of message queuing , We are more concerned about , At both ends of the message , I How do your business code work with message queuing , Achieve an optimal performance .

1. Send end performance optimization

The processing performance of the service code of the sender , It doesn't really have much to do with message queuing , Because the sender usually executes its own business logic first , Finally, send the message . if , The performance of your code to send messages is not good , You need to check it first , Is it the industry before the news is sent Business logic takes too much time .

For the business logic of sending messages , Just pay attention to setting the appropriate concurrency and batch size , Can achieve good transmission performance . What is this? What do you say ?

I said before Producer The process of sending messages ,Producer Send a message to Broker,Broker After receiving the message, return to confirm Respond to , It's a complete interaction . Suppose the average delay of this interaction is 1ms, We put this 1ms It's time to break down , It includes It's time-consuming :

• The sender prepares the data 、 Serialize message 、 Time to construct logic such as request , That is, the time consumed by the sender before sending the network request ;
• Time consuming of sending message and returning response in network transmission ;
• Broker Delay in processing messages .

If it's single threaded , Just send... At a time 1 Bar message , So you can only send 1000ms / 1ms * 1 strip /ms = 1000 strip news , such In this case, it doesn't give full play to the strength of message queue .

Whether it's increasing the batch size of each message sent , Or increase concurrency , Can double the transmission performance . As for whether to choose batch sending or increase concurrency , It mainly depends on the business nature of the sender program . Simply speaking , As long as it can meet your performance requirements , How to achieve convenience Just how to do it .

for instance , Your message sender is a micro service , Mainly accept RPC Request processing online business . Very natural , Microservice is processing, please When you ask , Just send the message directly in the current thread , Because all RPC Frameworks are multi-threaded and support multi concurrency , Nature is also real Now we can send messages in parallel . And the online business is more concerned about the request response delay , Choosing bulk delivery will inevitably affect RPC Service delay . This situation , The smarter way is to improve sending performance through concurrency .

If your system is an offline analysis system , What are the performance requirements of offline systems ? It doesn't care about delay , Pay more attention to the whole system throughput . The data of the sender comes from the database , This situation is more suitable for batch sending , You can batch read data from the database , however Send the message in bulk , Similarly, a very high throughput can be achieved with a small amount of concurrency .

2. Consumer performance optimization

When using message queuing , Most of the performance problems are on the consumer side , If the speed of consumption can't keep up with the speed of the sender producing messages , It creates a backlog of information . If this performance inversion problem is only temporary , That's not a big problem , As long as the performance of the consumer side is restored , exceed The performance of the sender , The backlog of information can be gradually digested .

If consumption has been slower than production , Time ⻓ 了 , The whole system will go wrong , or , The storage of the message queue is full and cannot be lifted For service , Or the message is lost , This is a serious failure for the whole system .

therefore , When we were designing the system , We must ensure that the consumption performance of the consumer side is higher than that of the production side , Such a system can be healthy Kang's continuous operation .

In addition to optimizing the logic of consumption business, the performance optimization of the consumer side , It can also be expanded horizontally , Increase the number of concurrent users to improve the overall consumption Cost performance . One particular thing to note is , It's expanding Consumer At the same time , The partitions in the expansion theme must be synchronized ( Also called queue ) Number , Make sure Consumer The number of instances is equal to the number of partitions .

If Consumer More instances than partitions , this This kind of expansion has no effect . The reason we talked about before , Because for consumers , In fact, only single thread consumption can be supported on each partition .

I ⻅ I've been to a lot of consumer programs , This is how they solve the problem of slow consumption :

 

The business logic of receiving message processing may be relatively slow , It's hard to optimize , To avoid message backlog , On receiving the message OnMessage In the method , Don't handle any business logic , Put this message in a memory queue and it returns .

And then it can start a lot of lines of business cheng , Inside these business threads are the real business logic for processing messages , These threads take messages from memory queues for processing , So it solves the problem of single individual Consumer The problem of not parallel consumption .

Is this method perfect for concurrent consumption ? Please note that , This is a very unusual situation ⻅ Error method of ! Why is it wrong ? Because it's going to lose news . If the node receiving the message goes down , These messages that have not been processed in the memory queue will be lost .

3、 ... and 、 How to deal with the backlog of messages ?

There is also a message backlog situation , When the daily system is working properly , There's no backlog, or only a small backlog, and it's quickly consumed , however At some point , All of a sudden, there's a backlog of news and the backlog keeps going up . In this case, you need to find out the reason for the message backlog in a short time , To solve the problem quickly will not affect the business .

There must be a variety of reasons for the sudden backlog , Different systems 、 Different situations have different reasons , Can't generalize . however , I We'll check the cause of the backlog , There are some relatively fixed and effective methods .

Can lead to a sudden increase in backlog , The coarsest reason , There are only two kinds : Or it's getting faster , Or consumption is slowing down .

Most message queues have built-in monitoring capabilities , Just by monitoring the data , It's easy to determine which cause . If it's sent per unit time More news , For example, catch up with big promotion or rush to buy , In a short time, it is impossible to optimize the code on the consumer side to improve the consumption performance , The only way It is to improve the overall consumption capacity by expanding the number of instances on the consumer side .

If there are not enough server resources to expand in a short time , The best way is , Demote the system , By closing down some unimportant businesses service , Reduce the amount of data sent by the sender , At least make the system work , Serve some important business .

There's another one that doesn't happen very often ⻅ The situation of , You're monitoring it to find that , No matter the speed of sending messages or consuming messages, it is the same as before turn , At this time, you need to check your consumer side , Is it a message caused by the failure of consumption? Repeated consumption is more common , This situation will also slow down the consumption speed of the whole system .

If we monitor that consumption is slowing down , You need to check your consumption examples , Analyze what causes consumption to slow down . Check the logs first Whether there are a lot of consumption mistakes , If there is no mistake , You can print stack information , Take a look at your consumption thread is not stuck in what The place doesn't move , For example, it triggers a deadlock or gets stuck waiting for some resources .

Four 、 Summary

Optimize messaging performance , There are two ways to prevent message backlog , Increasing batch or concurrency , At the sender, both methods can make use , On the consumer side, we need to pay attention to , Increasing the number of concurrent partitions requires synchronous expansion , Otherwise it won't work .

In case of message backlog in the system , We need to solve the backlog first , Analyze the reason again , After all, to ensure the availability of the system is the first problem to be solved topic . The fast way to solve the backlog is to increase it through horizontal expansion Consumer The number of instances .