Pulsar theme compression

Catalog

How topic compaction works

---

Pulsar Take highly scalable message data persistent storage as the main goal .Pulsar Topics enable you to persist the required number of unacknowledged messages , While preserving the message order . By default ,Pulsar Stores all unacknowledged or unprocessed messages generated on a topic . For many Pulsar scenario , It is necessary to store many unacknowledged messages on a topic , But for the Pulsar Users , In the log of the entire message “ The playback ” It will also become very time-consuming .

A more practical guide to topic compression , see also  Topic compaction cookbook.

For some scenarios , Consumers do not need the integrity of the topic log “ Images ”. They may Only a few values are needed to build more “ shallow ” Log image of , You may even need only the latest values . For these types of scenes ,Pulsar Provides theme compression . When you use compression on a topic ,Pulsar Will traverse the message backlog log of the topic , And remove the old message , in other words , It will press each key to traverse the topic , Only the latest news associated with the key is left .

Pulsar The theme compression function of ：

• Allows faster... Through topic logs “ The playback ”
• Only for persistent themes
• backlog It will automatically trigger when it reaches a certain size , It can also be triggered manually through the command line .See the Topic compaction cookbook
• Different from reservation in concept and operation （retention） Strategy and expiration （expiry） Strategy . Compared with the theme compression strategy , Retain （retention） Will give priority to . If retention Message deleted from message backlog for topic , Then the message will also not be able to be downloaded from the compressed subject ledger （ledger） Read from .

Topic compression example ： Stock market

Pulsar An example scenario for the compression theme is the stock market theme . In the stock symbol topic , The dollar value of each message with a timestamp , Used to buy shares （ The message key contains stock symbols , for example AAPL or GOOG）. For the stock market , You may only care about the latest value of the stock , Not interested in historical data （ namely , You don't need to build a complete image of the subject message sequence for each key ）. under these circumstances , Compression is very beneficial , Because it can avoid consumers' need to play back in confusing messages .

How topic compaction works

When passed CLI When topic compression is triggered （via the CLI）,Pulsar The whole topic will be traversed from beginning to end . For each Key, The compressor will retain this Key The latest record .// CLI :  Command line interface ( English :command-line interface, abbreviation :CLI

after ,broker A new ledger will be created （BookKeeper ledger）, And perform the second iteration for each message in the topic . For each message Key, If Key Match the latest data , Then Key Payload data for 、 news ID And metadata will be written to the newly created ledger （ledger）. If Key Does not match the latest data , The message will be skipped and kept . If the payload of any given message is empty （ For a message key Assign a null value ）, It will be skipped and treated as deleted （ Be similar to key-value In the database Tombstone concept -> That is, temporarily mark and delete ）. When the second iteration topic ends , The newly created ledger will be closed （BookKeeper ledger）, And write the following two contents into the metadata of the topic ： Ledger ID（BookKeeper ledger） And the last compressed message ID（ A compressed range called a topic compaction horizon）. Once this metadata is written , Compression is complete .

After the compression operation is initialized ,​ No matter when it comes to compressed ranges and compressed backlog messages （compacted backlog） Make any changes , Will inform the subject broker, such as ：

Client with read compression enabled （consumers and readers） Try to read a message from the topic , also ：

• Read the topic as normal （ If the news ID Greater than or equal to the compression range ） perhaps
• Read from the compressed range （ If the news ID Less than the compression range ）