当前位置:网站首页>Chapter 5 detailed explanation of consumer groups
Chapter 5 detailed explanation of consumer groups
2022-07-06 16:34:00 【Can't keep the setting sun】
5.1 Consumer group concept
What is? consumer group?
consumer group yes kafka The scalable and fault-tolerant consumer mechanism provided . Since it's a group , There must be multiple consumers or consumer instances in the group (consumer instance), They share a common ID, namely group ID. All consumers in the group coordinate to consume the theme of subscription (subscribed topics) All partitions of (partition). Each partition can only be made up of one... In the same consumption group consumer To consume .
consumer group Three characteristics
- consumer group There can be one or more consumer instance,consumer instance It can be a process , It can also be a thread
- group.id Is a string , A unique logo consumer group
- consumer group Subscribe to the topic Each partition of can only be assigned to one group Under a consumer( Of course, the partition can also be assigned to other group)
5.2 Consumer location
In the process of consumption, consumers need to record how much data they consume , Consumption location information . stay Kafka There is a special term for this location information in : Displacement (offset). Many message engines store this information on the server (broker End ). The advantage of this is that it is easy to realize , But there are three main problems :
- broker From then on, it becomes stateful , Will affect scalability ;
- Need to introduce a response mechanism (acknowledgement) To confirm the success of consumption .
- Because we need to save a lot consumer Of offset Information , It is inevitable to introduce complex data structures , Waste of resources .
and Kafka Choose a different way , Every consumer group Save your displacement information , Then only an integer is needed to represent the position ; At the same time, we can introduce checkpoint The mechanism is persistent on a regular basis , It simplifies the implementation of the response mechanism .
5.3 Displacement Management
5.3.1 Automatically VS Manual
Kafka The default is to automatically submit displacement reports periodically (enable.auto.commit = true
), Of course, you can choose to submit the displacement manually . in addition kafka I'll put it on a regular basis group Save the consumption , Make one offset map, As shown in the figure below
5.3.2 Displacement submission
The displacement of the old version is submitted to zookeeper in , The directory structure is /consumers/<group.id>/offsets/<topic>/<partitionId>
, however zookeeper In fact, it is not suitable for mass reading and writing operations , Especially write operations . therefore kafka Provides another solution : increase __consumer_offsets
topic, take offset Write this message topic, Get rid of right zookeeper Dependence ( Refers to preservation offset This matter ).__consumer_offsets The message in saves each consumer group Submitted at a certain moment offset Information . Still in the above figure consumer group For example , The format is something like this
边栏推荐
- Effet d'utilisation, déclenché lorsque les composants de la fonction sont montés et déchargés
- Research Report on market supply and demand and strategy of Chinese table lamp industry
- 顺丰科技智慧物流校园技术挑战赛(无t4)
- 2027. Minimum number of operations to convert strings
- Research Report of desktop clinical chemical analyzer industry - market status analysis and development prospect prediction
- Discussion on QWidget code setting style sheet
- 树莓派4B64位系统安装miniconda(折腾了几天终于解决)
- OneForAll安装使用
- Market trend report, technological innovation and market forecast of China's double sided flexible printed circuit board (FPC)
- MariaDB的安装与配置
猜你喜欢
Share an example of running dash application in raspberry pie.
Radar equipment (greedy)
< li> dot style list style type
Advancedinstaller installation package custom action open file
Hbuilder X格式化快捷键设置
Codeforces Round #801 (Div. 2)A~C
Pytorch extract skeleton (differentiable)
Raspberry pie 4B installation opencv3.4.0
(lightoj - 1323) billiard balls (thinking)
js封装数组反转的方法--冯浩的博客
随机推荐
(lightoj - 1354) IP checking (Analog)
(lightoj - 1349) Aladdin and the optimal invitation (greed)
业务系统从Oracle迁移到openGauss数据库的简单记录
树莓派4B64位系统安装miniconda(折腾了几天终于解决)
860. Lemonade change
The "sneaky" new asteroid will pass the earth safely this week: how to watch it
antd upload beforeUpload中禁止触发onchange
OneForAll安装使用
Research Report on hearing health care equipment industry - market status analysis and development prospect prediction
1529. Minimum number of suffix flips
Research Report on market supply and demand and strategy of Chinese table lamp industry
Input can only input numbers, limited input
Maximum product (greedy)
解决Intel12代酷睿CPU单线程只给小核运行的问题
解决Intel12代酷睿CPU单线程调度问题(二)
Problem - 922D、Robot Vacuum Cleaner - Codeforces
Tree of life (tree DP)
Investigation report of bench type Brinell hardness tester industry - market status analysis and development prospect prediction
Codeforces Round #803 (Div. 2)A~C
2027. Minimum number of operations to convert strings