Message queue data storage MySQL table design

1. Introduction

According to the requirements of the self-developed message queue, this article uses the MySQL database to design the storage of message queue message data.

The message queue supports multiple queue topics and uses the producer/consumer design pattern.

• Message producers can create queue topics and submit messages to custom queue topics.

• The message consumer pulls the message from the topic queue.

• The messages in the queue can be consumed only once or read by multiple consumers.

2. Message queue data table design

3. Key design

3.1 Use the same library table to store all topics

All queue topics are placed in the same logical data table (datatable is designed as above).

Two alternative designs are compared here:

• One table per queue theme

• All queue themes are placed in the same table

Design scheme one

The maximumThe problem is that message producers can create custom queue topics and create new tables for each queue topic, which is a waste of system resources.And it is not clear how many messages will be put in the queue topic. When the queue messages are read, whether to delete the table will become a big problem.

Design scheme 2

Put all messages in the same table, the above problems are solved, the increase and decrease of queue topics become the "addition and deletion" of the tableOperation, the operation is faster, and it is also easy to expand.We can index the queue topic to improve the speed of querying and reading messages.

This design choice

Design scheme two

.

3.2 Database Table Sharding

In terms of specific implementation, we can sharding the logical data table, create multiple tables with the same design, and put queue messages in multiple tables(such as message queue table 001, message queue table 002, message queue table 003, message queue table 004, ...).

Advantages

:

• Improve performance, store some messages in each table, and improve the speed of message reading and query.

• It is convenient for data clustering. When establishing a cluster, only some tables need to be moved to other servers.

• Convenient for data backup, you can back up updated data without re-backing up the entire table.

• It is convenient to discard old messages. Old messages that have been consumed can be deleted according to requirements, and the expired table can be deleted.

Sharding Algorithm

:

• Randomly store the message in the currently usedin the data sheet.

• Hash, calculate the hash value for the topic, and store the message in the corresponding data table according to the hash value.

Method 2 is simple to implement, just append a new message to the end of the current message queue table.But the disadvantage is that when you want to find a message, you have to read multiple sharding tables.This solution is more suitable for scenarios where there are far more write operations than read operations, or a scenario where there are only a few queue topics but many messages in the queue.

The implementation of the second method is more complicated, and the subject field of the new message needs to be hashed when reading and writing.The disadvantage is that both reads and writes are slower, but the advantage is that there is no need to read multiple tables.It is suitable for scenarios where there are far more read operations than write operations, or where users need to create a large number of queues, and each queue has fewer messages.

3.2 Index

Create an index on (queue topic, message sequence number), which is convenient to find all messages of a queue topic.