Chapter 5 detailed explanation of consumer groups

5.1 Consumer group concept

What is? consumer group?
consumer group yes kafka The scalable and fault-tolerant consumer mechanism provided . Since it's a group , There must be multiple consumers or consumer instances in the group (consumer instance), They share a common ID, namely group ID. All consumers in the group coordinate to consume the theme of subscription (subscribed topics) All partitions of (partition). Each partition can only be made up of one... In the same consumption group consumer To consume .

consumer group Three characteristics

• consumer group There can be one or more consumer instance,consumer instance It can be a process , It can also be a thread
• group.id Is a string , A unique logo consumer group
• consumer group Subscribe to the topic Each partition of can only be assigned to one group Under a consumer( Of course, the partition can also be assigned to other group)

5.2 Consumer location

In the process of consumption, consumers need to record how much data they consume , Consumption location information . stay Kafka There is a special term for this location information in ： Displacement (offset). Many message engines store this information on the server (broker End ). The advantage of this is that it is easy to realize , But there are three main problems ：

1. broker From then on, it becomes stateful , Will affect scalability ;
2. Need to introduce a response mechanism (acknowledgement) To confirm the success of consumption .
3. Because we need to save a lot consumer Of offset Information , It is inevitable to introduce complex data structures , Waste of resources .

and Kafka Choose a different way , Every consumer group Save your displacement information , Then only an integer is needed to represent the position ; At the same time, we can introduce checkpoint The mechanism is persistent on a regular basis , It simplifies the implementation of the response mechanism .

5.3 Displacement Management

5.3.1 Automatically VS Manual

Kafka The default is to automatically submit displacement reports periodically (enable.auto.commit = true), Of course, you can choose to submit the displacement manually . in addition kafka I'll put it on a regular basis group Save the consumption , Make one offset map, As shown in the figure below

5.3.2 Displacement submission

The displacement of the old version is submitted to zookeeper in , The directory structure is /consumers/<group.id>/offsets/<topic>/<partitionId>, however zookeeper In fact, it is not suitable for mass reading and writing operations , Especially write operations . therefore kafka Provides another solution ： increase __consumer_offsets topic, take offset Write this message topic, Get rid of right zookeeper Dependence ( Refers to preservation offset This matter ).__consumer_offsets The message in saves each consumer group Submitted at a certain moment offset Information . Still in the above figure consumer group For example , The format is something like this