Message Queuing Message Storage Design (Architecture Camp Module 8 Jobs)

Message table

For mysql, the performance of controlling the size of the table and reading and writing the table sequentially is the highest, so each topic is stored in a separate table, and eachThe table supports setting the maximum number of messages. If the maximum number of messages is exceeded, a new table will be created for storage.Message queues often have requirements for topic partition storage, so each partition also uses a separate table for storage.Therefore, the table name rule is:

message_{topic}_n_m

, where n is the partition serial number and m is the table serial number.For example, the business side creates a topic named test, the number of partitions is 4, and the maximum number of messages supported by each table is 100w. For the table of the first partition, the table name is message_test_0_0. When the first table exceeds 100w messagesAfter that, a second table will be created named message_test_0_1.

The id is set to primary key, and consumers passid reads the corresponding message.submit_time is set to a common index, which satisfies the need to clean up expired messages according to time.

Consume information table

consume_log table

A message table is not enough, and a table is needed to record the consumption status.There are also a few points:

• We support multiple consumer groups, each consumer group shares a consumption progress record, so the table needs to have a group field to distinguish which consumer isThe consumption record of the group;

• The consumption offset is the most important, which records the consumption data, which is identified by offset, that is, messageThe msg_id in the table, because msg_id is strictly increasing under one data group and one topic;

so the final message consumption record table is designed as follows:

If you don't consider too complicated design, the core is mainly these two tables.

Index Analysis

We first analyze several high-frequency scenarios:

Send messages

• CustomersThe end completes the shard routing, assigns it to a shard, and sends the message data

• The message queue server finds the corresponding message table according to the topic message_{topic}, and then insert data, msg_id auto-increment

Consume message

• The client completes the routing of the fragmentation and sends thePull the message from the corresponding server, bring the parameters topic, group

• to query the consumer_log table, find the corresponding data through the index according to topic, group, and take out the offset

• Query the corresponding message table according to the offset, find the next offset record through the primary key id, take it out and prepare to return

• Update the offset information of consumer_log

• return data

The above 2, 3, 4 are executed in one transaction.

It can be seen from 2 that message_log needs to add a topic+group composite index named idx_topic_group

It can be seen from 3 that the message table needs the index of msg_id, but it has alreadyis the primary key, no additional