Never use redis expired monitoring to implement scheduled tasks!

Preface

In the electricity supplier 、 Payment and other fields , There are often such scenes , The user gave up payment after placing an order , The order will be closed after the specified time period .

Careful, you must have found something like a treasure 、 A certain East has such logic , And the time is accurate , Error in 1s Inside , How did they achieve it ？

There are several general implementation methods ：

Use RocketMQ、RabbitMQ、Pulsar Wait for the delayed delivery function of the message queue
Use Redisson Provided DelayedQueue

There are some schemes that are widely circulated but have fatal defects , Don't use it to implement deferred tasks ：

Use Redis Expired monitoring for
Use RabbitMQ The dead letter line
Use non persistent time wheels

Redis Overdue monitoring

stay Redis The official Manual of keyspace-notifications: timing-of-expired-events It is clearly stated in ：

Basically expired events are generated when the Redis server deletes the key and not when the time to live theoretically reaches the value of zero

Redis The implementation of automatic expiration is ： Scheduled task offline scan and delete some expired keys ; Lazy checking for expiration when accessing keys and deleting expired keys .

Redis There is no guarantee that it will be deleted and sent an expiration notice at the set expiration time . actually , It is also common that the expiration notification is several minutes later than the set expiration time .

In addition, the key space notification is sent and forgotten （fire and forget） Strategy , Delivery is not guaranteed like message queuing . When a client subscribes to events, it will lose all the events distributed to it during disconnection .

This is a more “LOW” Solutions for , Please do not use .

RabbitMQ Dead letter

Dead letter （Dead Letter） yes RabbitMQ A mechanism provided .

When a message meets one of the following conditions, it will become dead letter ：

Message denied confirmation （ Such as channel.basicNack） And at this point requeue Property is set to false.
The message's lifetime in the queue exceeds the set TTL Time
The number of messages in the message queue has exceeded the maximum queue length

If dead letter queue is configured , Dead letter will be RabbitMQ Put it in the dead letter queue .

stay RabbitMQ The process of creating a dead letter queue in the ：

Create a switch as a dead letter switch
Configure in the business queue x-dead-letter-exchange and x-dead-letter-routing-key, Set the switch in the first step as the dead letter switch of the service queue
Create a queue on the dead letter switch , And listen to this queue

Dead letter queue is designed to store messages that are not normally consumed , Easy to check and re deliver . The dead letter queue also does not guarantee the delivery time , Before the first message becomes a dead letter , The following messages will not be delivered as dead letters even if they are expired .

To solve this problem ,Rabbit The official launch of the delayed delivery plug-in rabbitmq-delayed-message-exchange , It is recommended to use the official plug-in for delay messages .

Here's a digression , Use Redis Overdue monitoring or RabbitMQ The dead letter queue uses Middleware in a way unexpected to the designer , There are some hidden dangers in this unexpected behavior , Such as lack of consistency and reliability assurance , Low throughput 、 Resource leakage, etc .

A famous example is that many people use Redis Of List As message queue , So that the last author can't read and write Disque And finally evolved into Redis Stream. Try not to abuse middleware at work , Do professional things with professional components .

Time wheel

Time wheel is an excellent data structure for timed tasks , However, the vast majority of time wheel implementations are pure memory without persistence .

After the process running the time wheel crashes, all the tasks in it will disappear , So I advise you to use it carefully .

Redisson DelayQueue

Redisson DelayQueue It's based on Redis Zset Implementation of delay queue based on .DelayQueue One of them is called timeoutSetName Ordered set of , Of the elements score Is the delivery timestamp .

DelayQueue Can use regularly zrangebyscore Scan messages that have reached the delivery time , Then move them to the ready message list .

DelayQueue Guarantee Redis Messages are not lost without crashing , Try it when you don't have a better solution .

When the database index is well designed , The overhead of regularly scanning the database for outstanding orders is not as large as expected .

In the use of Redisson DelayQueue The method of scanning the database can be used as the compensation mechanism when waiting for timed task middleware , Avoid task loss caused by middleware failure .

Conclusion

The following points are summarized ：

First of all, it is recommended to use RocketMQ、Pulsar And other message queues with scheduled delivery function .
When it is not convenient to obtain a professional message queue, you can consider using Redisson DelayQueue Based on Redis Delay queue scheme , But for Redis Design compensation protection mechanism in case of crash .
When you can't use Redisson DelayQueue You can consider using the time wheel . Because the time wheel restart is far better than Redis Restart frequently , Protection mechanisms such as regular library scanning are more important .
Never use Redis Overdue monitoring implements scheduled tasks .

The source code attachment has been packaged and uploaded to Baidu cloud , You can download it yourself ～

link : https://pan.baidu.com/s/14G-bpVthImHD4eosZUNSFA?pwd=yu27
Extraction code : yu27
Baidu cloud link is unstable , It may fail at any time , Let's keep it tight .

If Baidu cloud link fails , Please leave me a message , When I see it, I will update it in time ～

Open source address

Code cloud address ：
http://github.crmeb.net/u/defu

Github Address ：
http://github.crmeb.net/u/defu