Redis message queue

List Implement message queuing

Redis List is a simple list of strings , Sort by insertion order . You can add an element to the head of the list ( On the left ) Or tail ( On the right ).

So it is often used for asynchronous queues . Sequence the task structures that need to be delayed into strings and insert them into Redis A list of , Another thread polls data from this list for processing .

 

The problem of instant consumption

adopt LPUSH,RPOP The way , There is a performance risk , If consumers want to process data in time , We have to write something similar in the program while(true) Such logic , Keep calling RPOP or LPOP command , This will bring some unnecessary performance loss to consumer programs .

therefore ,Redis It also provides BLPOP、BRPOP This block read command ( belt B-Bloking It's all blocking ), When the client does not read the queue data , Automatic blocking , Until new data is written to the queue , Start reading new data again . This way saves unnecessary CPU expenses .

• LPUSH、BRPOP Left in, right out
• RPUSH、BLPOP Right in and left out

If the timeout is set to 0 when , You can wait forever , Until a message pops up

because Redis Characteristics of single thread , So when it comes to consumption data , Can the same message be shared by multiple users at the same time consumer Consumption , But we need to consider the situation of unsuccessful consumption .

ack Mechanism

List Once the message in the queue is sent , Then delete it from the queue . If the consumer doesn't receive a message due to the Internet , Or the consumer crashes in the process of processing this message , You can't restore this message any more . The reason is , It's the lack of message confirmation mechanism .

To ensure the reliability of the message , Message queue has perfect message confirmation mechanism (Acknowledge), That is, the mechanism by which consumers report to the queue that a message has been received or processed .

 

 RPOPLPUSH、BRPOPLPUSH ( Blocking ) From a list Copy the message to another while getting the message in list in ( It can be used as a backup ), And this process is atomic .

So that when the business process is secure , Delete the queue element , Implement message confirmation mechanism .

Subscription and publication implement message queue

 

" Release / subscribe " There are two roles in the pattern , Publishers and subscribers . Subscribers can subscribe to one or more channels (channel), And publishers can send messages to specified channels (channel) Send a message , All subscribers who subscribe to this channel will receive this message .

Redis adopt PUBLISH 、 SUBSCRIBE And other commands realize subscription and publishing ,Redis Publish subscribe (pub/sub) One drawback is that messages don't persist , If there is a network disconnect 、Redis Downtime, etc , The message will be discarded . And no Ack Mechanism to ensure the reliability of data , Suppose a consumer doesn't have , Then the message is discarded .

Streams Implement message queuing

Redis 5.0 The version adds a more powerful data structure ——Stream. It provides message persistence and primary / standby replication , Any client can access data at any time , And remember the location of each client's access , It also ensures that the message is not lost .

It is like a message linked list with only additional content , String all the added messages together , Every message has a unique ID And the corresponding content . And the message is persistent .

 

Every Stream All have unique names , It is Redis Of key, In our first use xadd Automatically create when the command appends a message .

Redis Stream I learned a lot from Kafka The design of the .

• Consumer Group： With the concept of consumer groups , Each consumer group status is independent , They don't influence each other , A consumer group can have multiple consumers
• last_delivered_id ： Each consumer group will have a cursor last_delivered_id Move forward over the array , Indicates which message has been consumed by the current consumption group
• pending_ids ： Consumer state variables , The function is to protect consumers' unconfirmed id.pending_ids The message that has been read by the client is recorded , But not yet ack. If the client does not ack, The message in this variable ID More and more , Once a message is ack, It starts to decrease . This pending_ids Variable in Redis Officially it's called PEL, That is to say Pending Entries List, This is a very core data structure , It is used to ensure that the client consumes the message at least once , And will not be lost in the middle of the network transmission .

 

Stream Unlike Kafak So there's the concept of zoning , If you want to achieve a function similar to partition , It is necessary to use a certain strategy in the client to write messages to different Stream.

• xgroup create： Create consumer groups
• xgreadgroup： Read messages from consumer groups
• xack：ack Drop the specified message

Consumption by consumption group

Stream Provides xreadgroup Instruction can be used for intra group consumption of consumption group , You need to provide the consumer group name 、 Consumer name and start message ID. It is the same as xread equally , It can also block waiting for new messages . After reading the new news , The corresponding message ID Will enter the consumer's PEL( The message being processed ) In structure , Use after client processing xack The command notifies the server , This message has been processed , This message ID It will start from PEL Remove .

Stream It's still not mainstream MQ To use the , And there are fewer use cases , Use with caution .