MySQL table design for message queue to store message data

Analysis of message queue storage structure

message table

First of all, we need a table to store the message body, we tentatively set it as the message table.There are several points:

• Because the data is grouped in the design of the system architecture, a field representing the grouping is required in the table: shard_id.The shard_id of each group of data grouping clusters is fixed.This is similar to the partition design of kafka, different consumers can pull different shards to achieve the purpose of accelerating consumption;

• Here design eachEach topic is stored in a separate table, that is,
  message_{topic}
  , the reason is 1. MySQL theoretically supports hundreds of millions of tables, which can meet the number of topics required 2.Using a single table can isolate the impact of each topic (physically and logically). 3. It is relatively simple to use, because only the messages of the topic can be stored in a single table, and the id can also be auto-incremented.

So the final message table is designed as follows:

consume_log table

It is not enough to have a message table, but also a table to record the consumption status.There are also a few points:

• We support multiple consumer groups, each consumer group shares a consumption progress record, so the table needs to have a group field to distinguish which consumer isThe consumption record of the group;

• The consumption offset is the most important, which records the consumption data, which is identified by offset, that is, messageThe msg_id in the table, because msg_id is strictly increasing under one data group and one topic;

So the final message consumption record table is designed as follows:

If you don't consider too complicated design, the core is mainly these two tables.

Index Analysis

We first analyze several high-frequency scenarios:

Send messages

• CustomersThe end completes the shard routing, assigns it to a shard, and sends the message data

• The message queue server finds the corresponding message table message_{topic} according to the topic, and then insert data, msg_id is incremented automatically

Consume message

• The client completes the routing of the fragmentation and sends thePull the message from the corresponding server, bring the parameters topic, group

• to query the consumer_log table, find the corresponding data through the index according to the topic and group, and take out the offset

• Query the corresponding message table according to the offset, find the record of the next offset through the primary key id, take it out and prepare to return

• Update the offset information of consumer_log

• Return data

The above 2,3,4 are executed in one transaction.

As can be seen from 2, message_log needs to add a topic+group combination index named idx_topic_group

As can be seen from 3, the message table needs the index of msg_id, butIt is already the primary key, so there is no need to add additional

message expiration

1. The scheduled task starts regularly (such as 1 day), calculates the date value that needs to be cleaned, and scans the message table, find the records that need to expire, and the batch update status is deleted (or directly physically deleted)

It can be seen here that in order to quickly scan the records to be expired, the born_date of the message needs to be indexed.Named idx_born_date

Index summary

So according to the analysis, the following indexes are needed:

• message_log table added
  idx_topic_group
  , is the combined index of topic+group

• message table added
  idx_born_date
  , is the index of born_date