New generation cloud native message queue (II)

Write it at the front

  The last one mainly introduced Pulsar The overall architecture and source directory structure , It is the basis for us to analyze the source code . This article , Let's talk about the core concept of message queue --Topic,Pulsar How is it realized .

Topic brief introduction

 Topic Is the core concept of message queue , Messages sent by producers will be saved in Topic in . When sending a message, the producer can specify which one to send Topic, Consumers also take Topic Dimension of subscribes to and consumes a Topic Next news .

 Pulsar Medium Topic The classification is quite detailed , According to the storage form, it is divided into ordinary Topic And zoning Topic. Ordinary Topic The news will only fall on one Broker On , And single Broker The performance of is always limited , So ordinary Topic The message throughput is not high . that , Partition is required Topic, Each partition is in a Broker On , You can make multiple Broker Carrying capacity .

  Depending on whether the message is persistent , It is also divided into persistence Topic And non persistent Topic, The details will be introduced later .

 Pulsar in Topic The format of ：

{persistent | non-persistent}://tenant/namespace/topic

 Topic Will persistent perhaps non-persistent start , The former shows that this is a persistence Topic, When a message is sent to this Topic after , Messages will be persisted and stored in Bookie in ; The latter indicates that messages will only be saved in Broker The memory of the , When Broker Restart or crash , The news will be lost .

Topic Once created , The persistent type can no longer be modified

.

 Topic The second part of is the tenant name ; The third part is the namespace name ; The fourth part is the definition Topic name . 

 Topic Can automatically create , You can also call the management flow API Create . Automatic creation needs to be done in Broker Configuration file for broker.conf Set two parameters in ：

allowAutoTopicCreation = true
# non-partitioned  or  partitioned
allowAutoTopicCreationType = non-partitioned 

  As long as it's set allowAutoTopicCreation = true , Will be creating Producer、Consumer When , If Topic If it doesn't exist, it will be created automatically . therefore , To prevent Topic flood , Usually turn off automatic creation Topic, And by managing flow API Manually create .

  Ordinary Topic And zoning Topic Cannot convert each other , You can create a single partition partition Topic, Easy to expand later .

Topic The type of

 Topic There are four kinds of , They are persistent partitions Topic、 Persistent nonpartitioned Topic、 Non persistent partitions Topic And nonpersistent nonpartitioned Topic.

  Persistence Topic And non persistent Topic The core difference is that the data is persistent to Bookie in , Or stored in the current Broker The memory of the . So persistence Topic When it's new , It will be recorded Ledger Information , Record this Partition To which Bookie In nodes . And zoning Topic And non partition Topic The core difference is Topic Whether metadata information of needs to be synchronized . So zoning Topic At creation time , I'll put Topic Metadata information is written into the corresponding metadata storage ( such as Zookeeper、etcd etc. )

Topic The core

 Pulsar Of Topic Both need and Broker binding , Non partitioned Topic Bind single Broker, The partition Topic Each Partition Bind single Broker. that Topic and Broker How to build a relationship ？

  as everyone knows ,

Pulsar Can be millions Topic, If you store each in the metadata store Topic and Broker The relationship between , Then the amount of data is huge . therefore Pulsar It's not about Topic and Broker Directly bound , But abstract Bundle The concept of , And use consistency Hash Algorithm .Bundle As a virtual node in a consistent hash ring ,Topic Calculate hash value by name , And hash into the hash ring , Find the corresponding Bundle, Then map to the corresponding Broker On .

 

that Topic Specifically, how to map , How does the source code operate . Next section Pulsar Explain in detail .