当前位置：网站首页>译文推荐｜Apache Pulsar 隔离系列（四）：单集群隔离策略
译文推荐｜Apache Pulsar 隔离系列（四）：单集群隔离策略
2022-08-04 18:43:00 【StreamNative】
本文翻译自 StreamNative 博客《Pulsar Isolation Part IV: Single Cluster Isolation》[1]，作者高冉，StreamNative 软件工程师，Apache Pulsar Committer。
校对：StreamNative
译者简介
宋博，就职于北京百观科技有限公司，高级开发工程师，专注于微服务、云计算与大数据领域。
介绍
这是我们关于如何在 Apache Pulsar 中实现资源隔离博客系列中的第四篇。在深入研究之前，让我们回顾一下前三篇博客所涵盖的内容。
深度解析如何在 Pulsar 中实现隔离介绍了在 Pulsar 中实现隔离的三种方法。包括：
• 使用独立的 BookKeeper 集群和独立的 Pulsar 集群
• 使用独立的 Pulsar 集群共享一个 BookKeeper 集群
• 使用单个的 Pulsar 集群和单个的 BookKeeper 集群。在本系列后续的博客中详细讨论了每种方法及其特定场景。
零经验玩转隔离策略：多个 Pulsar 集群演示如何使用独立 BookKeeper 集群在各个 Pulsar 集群之间实现隔离。这种无共享的方法提供了最高级别的隔离，适用于存储高度敏感的数据，例如个人身份信息或财务记录。
Pulsar 独立集群如何实现单一、共享 BookKeeper演示多个独立的 Pulsar 集群如何在共享一个 BookKeeper 集群的情况下实现 Pulsar 隔离。这种方法使用单独的 Pulsar broker 集群来隔离最终用户，并允许你根据场景使用不同的身份验证方法。存储层相互共享也有多种好处，例如可以减少对硬件的依赖，相关的硬件和运维成本也更低。
在本系列的第四篇也是最后一篇博客中，我们将逐步演示如何使用单个集群实现 broker 和 bookie 的隔离。这种更传统的方法利用了 Pulsar 内置的多租户特性，并且无需管理多个 broker 和 bookie 集群。
准备工作
本教程将使用 docker-compose 来建立 Pulsar 集群。我们先需要安装 Docker 环境[2]。
本教程基于 Docker 20.10.10、docker-compose 1.29.2 和 macOS 12.3.1。
1. 获取 docker-compose 配置文件。
git clone https://github.com/gaoran10/pulsar-docker-compose
cd pulsar-docker-compose2. 启动集群。
docker-compose up -d3. 查看 Pod 状态。
docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------
bk1 bash -c export dbStorage_w ... Up
bk2 bash -c export dbStorage_w ... Up
bk3 bash -c export dbStorage_w ... Up
bk4 bash -c export dbStorage_w ... Up
broker1 bash -c bin/apply-config-f ... Up
broker2 bash -c bin/apply-config-f ... Up
broker3 bash -c bin/apply-config-f ... Up
proxy1 bash -c bin/apply-config-f ... Up 0.0.0.0:6650->6650/tcp, 0.0.0.0:8080->8080/tcp
pulsar-init bin/init-cluster.sh Exit 0
zk1 bash -c bin/apply-config-f ... Up集群初始化完成后，我们可以开始设置 broker 隔离策略。
Broker 隔离
1. 下载 Pulsar 安装包并解压。
wget https://archive.apache.org/dist/pulsar/pulsar-2.10.0/apache-pulsar-2.10.0-bin.tar.gz
tar -txvf apache-pulsar-2.10.0-bin.tar.gz
// 在此目录下执行 pulsar-admin 命令
cd apache-pulsar-2.10.02. 获取 broker 列表。
bin/pulsar-admin brokers list test
# 输出
"broker1:8080"
"broker2:8080"
"Broker3:8080"3. 创建一个 namespace。
bin/pulsar-admin namespaces create public/ns-isolation
bin/pulsar-admin namespaces set-retention -s 1G -t 3d public/ns-isolation4. 设置 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy set \
--auto-failover-policy-type min_available \
--auto-failover-policy-params min_limit=1,usage_threshold=80 \
--namespaces public/ns-isolation \
--primary "broker1:*" \
--secondary "broker2:*" \
test ns-broker-isolation5. 查看 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy list test
# 输出
ns-broker-isolation NamespaceIsolationDataImpl(namespaces=[public/ns-isolation], primary=[broker1:*], secondary=[broker2:*], autoFailoverPolicy=AutoFailoverPolicyDataImpl(policyType=min_available, parameters={min_limit=1, usage_threshold=80}))6. 创建一个分区主题。
bin/pulsar-admin topics create-partitioned-topic -p 10 public/ns-isolation/t17. 查看分区主题所属的 broker。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker1:6650
8. 停止 broker1。
${DOCKER_COMPOSE_HOME}/docker-compose stop broker1
# 输出
Stopping broker1 ... done9. 再次查看分区主题所属 broker。当 broker 1 停止服务后，原主题将归备用y broker
broker2:*所有。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker2:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker2:665010. 停止 broker2。
${DOCKER_COMPOSE_HOME}/docker-compose stop broker2
# 输出
Stopping broker2 ... done11. 查看分区主题所属的 broker。
当停止 broker2 后，namespace public/ns-isolation-broker 就没有可用的 broker 了。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
HTTP 503 Service Unavailable
Reason: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable12. 重启 broker1 和 broker2。
${DOCKER_COMPOSE_HOME}/docker-compose start broker1
# 输出
Starting broker1 ... done
${DOCKER_COMPOSE_HOME}/docker-compose start broker2
# 输出
Starting broker2 ... done在 Broker 之间迁移 Namespace
1. 检查 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy list test
# 输出
ns-broker-isolation NamespaceIsolationDataImpl(namespaces=[public/ns-isolation], primary=[broker1:*], secondary=[broker2:*], autoFailoverPolicy=AutoFailoverPolicyDataImpl(policyType=min_available, parameters={min_limit=1, usage_threshold=80}))我们可以看到 namespace
public/ns-isolation的 primary 和 secondary 的 broker 分别是broker1:*和broker2:*。2. 查看该分区主题所属 broker。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker1:66503. 修改新的 namespace 隔离策略。
bin/pulsar-admin ns-isolation-policy set \
--auto-failover-policy-type min_available \
--auto-failover-policy-params min_limit=1,usage_threshold=80 \
--namespaces public/ns-isolation \
--primary "broker3:*" \
--secondary "broker2:*" \
test ns-broker-isolation4. 查看 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy list test
# 输出
ns-broker-isolation NamespaceIsolationDataImpl(namespaces=[public/ns-isolation], primary=[broker3:*], secondary=[broker2:*], autoFailoverPolicy=AutoFailoverPolicyDataImpl(policyType=min_available, parameters={min_limit=1, usage_threshold=80}))5. 卸载该 namespace 来让该 namespace 的隔离策略生效。
bin/pulsar-admin namespaces unload public/ns-isolation6. 查看分区主题的所属 broker。我们可以看到主题的所属 broker 已更改为 primary broker broker3。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker3:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker3:6650扩容和缩容 broker
扩容
1. 启动 broker4。
在 docker-compose 文件中添加 broker4 配置。
broker4:
hostname: broker4
container_name: broker4
image: apachepulsar/pulsar:latest
restart: on-failure
command: >
bash -c "bin/apply-config-from-env.py conf/broker.conf && \
bin/apply-config-from-env.py conf/pulsar_env.sh && \
bin/watch-znode.py -z $$zookeeperServers -p /initialized-$$clusterName -w && \
exec bin/pulsar broker"
environment:
clusterName: test
zookeeperServers: zk1:2181
configurationStore: zk1:2181
webSocketServiceEnabled: "false"
functionsWorkerEnabled: "false"
managedLedgerMaxEntriesPerLedger: 100
managedLedgerMinLedgerRolloverTimeMinutes: 0
volumes:
- ./apply-config-from-env.py:/pulsar/bin/apply-config-from-env.py
depends_on:
- zk1
- pulsar-init
- bk1
- bk2
- bk3
- bk4
networks:
pulsar:启动 broker4。
${DOCKER_COMPOSE_HOME}/docker-compose create
# 输出
zk1 is up-to-date
bk1 is up-to-date
bk2 is up-to-date
bk3 is up-to-date
broker1 is up-to-date
broker2 is up-to-date
broker3 is up-to-date
Creating broker4 ... done
proxy1 is up-to-date
${DOCKER_COMPOSE_HOME}/docker-compose start broker4
# 输出
Starting broker4 ... done2. 查看 broker 列表。
bin/pulsar-admin brokers list test
# 输出
broker4:8080
broker1:8080
broker2:8080
broker3:80803. 设置 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy set \
--auto-failover-policy-type min_available \
--auto-failover-policy-params min_limit=1,usage_threshold=80 \
--namespaces public/ns-isolation \
--primary "broker1:*,broker4:*" \
--secondary "broker2:*" \
test ns-broker-isolation4. 获取 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy list test
# 输出
ns-broker-isolation NamespaceIsolationDataImpl(namespaces=[public/ns-isolation], primary=[broker1:*, broker4:*], secondary=[broker2:*], autoFailoverPolicy=AutoFailoverPolicyDataImpl(policyType=min_available, parameters={min_limit=1, usage_threshold=80}))5. 卸载 namespace。
bin/pulsar-admin namespaces unload public/ns-isolation6. 查看分区主题所属 broker。该主题应该同时归属于 broker1 和 broker4。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker4:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker4:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker4:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker4:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker1:6650缩容
1. 将 broker4 从 namespace 的隔离策略中移除。
bin/pulsar-admin ns-isolation-policy set \
--auto-failover-policy-type min_available \
--auto-failover-policy-params min_limit=1,usage_threshold=80 \
--namespaces public/ns-isolation \
--primary "broker1:*" \
--secondary "broker2:*" \
test ns-broker-isolation2. 查看 namespace 的隔离策略。
bin/pulsar-admin ns-isolation-policy list test
# 输出
ns-broker-isolation NamespaceIsolationDataImpl(namespaces=[public/ns-isolation], primary=[broker1:*], secondary=[broker2:*], autoFailoverPolicy=AutoFailoverPolicyDataImpl(policyType=min_available, parameters={min_limit=1, usage_threshold=80}))3. 停止 broker4。
${DOCKER_COMPOSE_HOME}/docker-compose stop broker4
# 输出
Stopping broker4 ... done4. 查看 broker 列表。
bin/pulsar-admin brokers list test
# 输出
broker1:8080
broker2:8080
Broker3:80805. 查看分区主题所属 broker。
bin/pulsar-admin topics partitioned-lookup public/ns-isolation/t1
# 输出
persistent://public/ns-isolation/t1-partition-0 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-1 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-2 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-3 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-4 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-5 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-6 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-7 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-8 pulsar://broker1:6650
persistent://public/ns-isolation/t1-partition-9 pulsar://broker1:6650BookKeeper 隔离
1. 获取 bookie 列表。
bin/pulsar-admin bookies list-bookies
# 输出
{
"bookies" : [ {
"bookieId" : "bk2:3181"
}, {
"bookieId" : "bk4:3181"
}, {
"bookieId" : "bk3:3181"
}, {
"bookieId" : "bk1:3181"
} ]
}2. 设置 bookie rack。
bookkeeperClientRackawarePolicyEnabled 的默认配置值是 true，所以 RackawareEnsemblePlacementPolicy 是 bookie 的默认隔离策略，我们将设置 rack 的名称如下：/rack。
bin/pulsar-admin bookies set-bookie-rack \
--bookie bk1:3181 \
--hostname bk1:3181 \
--group group1 \
--rack /rack1
bin/pulsar-admin bookies set-bookie-rack \
--bookie bk3:3181 \
--hostname bk3:3181 \
--group group1 \
--rack /rack1
bin/pulsar-admin bookies set-bookie-rack \
--bookie bk2:3181 \
--hostname bk2:3181 \
--group group2 \
--rack /rack2
bin/pulsar-admin bookies set-bookie-rack \
--bookie bk4:3181 \
--hostname bk4:3181 \
--group group2 \
--rack /rack23. 查看 bookie rack 配置（机架配置）。
bin/pulsar-admin bookies racks-placement
group1 {bk1:3181=BookieInfoImpl(rack=/rack1, hostname=bk1:3181), bk3:3181=BookieInfoImpl(rack=/rack1, hostname=bk3:3181)}
group2 {bk2:3181=BookieInfoImpl(rack=/rack2, hostname=bk2:3181), bk4:3181=BookieInfoImpl(rack=/rack2, hostname=bk4:3181)}4. 为 namespace 设置 bookie affinity group（关联组）。
bin/pulsar-admin namespaces set-bookie-affinity-group public/ns-isolation \
--primary-group group1 \
--secondary-group group25. 查看 namespace affinity group。
bin/pulsar-admin namespaces get-bookie-affinity-group public/ns-isolation
{
"bookkeeperAffinityGroupPrimary" : "group1",
"bookkeeperAffinityGroupSecondary" : "group2"
}6. 向该主题生产消息。
bin/pulsar-client produce -m 'hello' -n 500 public/ns-isolation/t27. 查看该主题的内部状态。
bin/pulsar-admin topics stats-internal public/ns-isolation/t2 | grep ledgerId | tail -n 6
"ledgerId" : 0,
"ledgerId" : 1,
"ledgerId" : 2,
"ledgerId" : 3,
"ledgerId" : 4,
"ledgerId" : -1,8. 查看 ledger [0, 1, 2, 3, 4] 的 ledger ensembles。
# 在 bk1 节点上执行以下命令
${DOCKER_COMPOSE_HOME}/docker-compose exec bk1 /bin/bash
bin/bookkeeper shell ledgermetadata -ledgerid 0
# 查看 ensembles
ensembles={0=[bk1:3181, bk3:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 1
# 查看 ensembles
ensembles={0=[bk3:3181, bk1:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 2
# 查看 ensembles
ensembles={0=[bk1:3181, bk3:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 3
# 查看 ensembles
ensembles={0=[bk1:3181, bk3:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 4
# 查看 ensembles
ensembles={0=[bk1:3181, bk3:3181]}9. 停止 bookie1。
${DOCKER_COMPOSE_HOME}/docker-compose stop bk110. 向该主题生产消息。
bin/pulsar-client produce -m 'hello' -n 500 public/ns-isolation/t211. 查看 ledger 的元数据。
bin/pulsar-admin topics stats-internal public/ns-isolation/t2 | grep ledgerId | tail -n 6
"ledgerId" : 5,
"ledgerId" : 6,
"ledgerId" : 7,
"ledgerId" : 8,
"ledgerId" : 9,
"ledgerId" : -1,查看新增 ledger [5, 6, 7, 8, 9] 的元数据。bookkeeperClientEnforceMinNumRacksPerWriteQuorum 的配置为 false 且 bookie1 始终不可用，由此可知会使用 secondary bookie。bookie3 位于 primary group 中，因此始终使用 bookie3。
# 在 bk2 节点上执行以下命令${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 /bin/bash
bin/bookkeeper shell ledgermetadata -ledgerid 5
# 查看 ensemble
ensembles={0=[bk4:3181, bk3:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 6
# 查看 ensemble
ensembles={0=[bk3:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 7
# 查看 ensemble
ensembles={0=[bk2:3181, bk3:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 8
# 查看 ensembles=
ensembles={0=[bk3:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 9
# 查看 ensemble
ensembles={0=[bk3:3181, bk2:3181]}重启 bk1。
${DOCKER_COMPOSE_HOME}/docker-compose start bk1迁移 Bookie Affinity Group
1. 查看 bookie affinity group。
bin/pulsar-admin namespaces get-bookie-affinity-group public/ns-isolation
{
"bookkeeperAffinityGroupPrimary" : "group1",
"bookkeeperAffinityGroupSecondary" : "group2"
}2. 修改 namespace 的 bookie affinity group。
bin/pulsar-admin namespaces set-bookie-affinity-group public/ns-isolation \
--primary-group group23. 查看 bookie affinity group。
bin/pulsar-admin namespaces get-bookie-affinity-group public/ns-isolation
{
"bookkeeperAffinityGroupPrimary" : "group2"
}4. 卸载 namespace。
bin/pulsar-admin namespaces unload public/ns-isolation5. 生产消息。
bin/pulsar-client produce -m 'hello' -n 500 public/ns-isolation/t26. 查看 ensemble 的各个 bookie 上是否有新创建的 ledger。
bin/pulsar-admin topics stats-internal public/ns-isolation/t2 | grep ledgerId | tail -n 6
"ledgerId" : 12,
"ledgerId" : 13,
"ledgerId" : 14,
"ledgerId" : 15,
"ledgerId" : 16,
"ledgerId" : -1,7. 查看新添加的 ledger [12, 13, 14, 15, 16] 的元数据。
# 在 bk2 节点上执行以下命令
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 /bin/bash
bin/bookkeeper shell ledgermetadata -ledgerid 12
# 查看 ensemble
ensembles={0=[bk4:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 13
# 查看 ensemble
ensembles={0=[bk4:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 14
# 查看 ensemble
ensembles={0=[bk4:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 15
# 查看 ensemble
ensembles={0=[bk4:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 16
# 查看 ensemble
ensembles={0=[bk2:3181, bk4:3181]}扩容和缩容 bookie
扩容
1.在 docker-compose 文件中追加以下配置。
bk5:
hostname: bk5
container_name: bk5
image: apachepulsar/pulsar:latest
command: >
bash -c "export dbStorage_writeCacheMaxSizeMb="${dbStorage_writeCacheMaxSizeMb:-16}" && \
export dbStorage_readAheadCacheMaxSizeMb="${dbStorage_readAheadCacheMaxSizeMb:-16}" && \
bin/apply-config-from-env.py conf/bookkeeper.conf && \
bin/apply-config-from-env.py conf/pulsar_env.sh && \
bin/watch-znode.py -z $$zkServers -p /initialized-$$clusterName -w && \
exec bin/pulsar bookie"
environment:
clusterName: test
zkServers: zk1:2181
numAddWorkerThreads: 8
useHostNameAsBookieID: "true"
volumes:
- ./apply-config-from-env.py:/pulsar/bin/apply-config-from-env.py
depends_on:
- zk1
- pulsar-init
networks:
pulsar:2. 启动 bookie5。
${DOCKER_COMPOSE_HOME}/docker-compose create
${DOCKER_COMPOSE_HOME}/docker-compose start bk53. 检查可读可写的 bookie 列表。因为 bookie1 已停止，所以应该有 4 个 bookie。
# 在 bk2 上执行此命令
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 bin/bookkeeper shell listbookies -rw
ReadWrite Bookies :
BookieID:bk2:3181, IP:192.168.32.5, Port:3181, Hostname:bk2
BookieID:bk4:3181, IP:192.168.32.7, Port:3181, Hostname:bk4
BookieID:bk3:3181, IP:192.168.32.6, Port:3181, Hostname:bk3
BookieID:bk1:3181, IP:192.168.32.4, Port:3181, Hostname:bk1
BookieID:bk5:3181, IP:192.168.32.9, Port:3181, Hostname:bk54. 将新添加的 bookie 节点添加到 primary group 中。
bin/pulsar-admin bookies set-bookie-rack \
--bookie bk5:3181 \
--hostname bk5:3181 \
--group group2 \
--rack /rack25. 检查 bookie rack 配置。
bin/pulsar-admin bookies racks-placement
group1 {bk1:3181=BookieInfoImpl(rack=/rack1, hostname=bk1:3181), bk3:3181=BookieInfoImpl(rack=/rack1, hostname=bk3:3181)}
group2 {bk2:3181=BookieInfoImpl(rack=/rack2, hostname=bk2:3181), bk4:3181=BookieInfoImpl(rack=/rack2, hostname=bk4:3181), bk5:3181=BookieInfoImpl(rack=/rack2, hostname=bk5:3181)}6. 卸载 namespace。
bin/pulsar-admin namespaces unload public/ns-isolation7. 向新主题生产消息。
bin/pulsar-client produce -m 'hello' -n 500 public/ns-isolation/t28. 查看该主题中新添加的 ledger。
bin/pulsar-admin topics stats-internal public/ns-isolation/t2 | grep ledgerId | tail -n 6
"ledgerId" : 17,
"ledgerId" : 20,
"ledgerId" : 21,
"ledgerId" : 22,
"ledgerId" : 23,
"ledgerId" : -1,通过查看 ledger 的 ensemble，我们可以发现新建的 ledger 都写入了 primary group 中的节点，因为有足够的 rw bookie 可用。
# 在 bk1 节点上执行以下命令
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 /bin/bash
bin/bookkeeper shell ledgermetadata -ledgerid 17
# 查看 ensemble
ensembles={0=[bk5:3181, bk2:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 20
# 查看 ensemble
ensembles={0=[bk2:3181, bk4:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 21
# 查看 ensemble
ensembles={0=[bk5:3181, bk4:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 22
# 查看 ensemble
ensembles={0=[bk5:3181, bk4:3181]}
bin/bookkeeper shell ledgermetadata -ledgerid 23
# 查看 ensemble
ensembles={0=[bk2:3181, bk4:3181]}缩容
1. 查看 rack 配置。
bin/pulsar-admin bookies racks-placement
group1 {bk1:3181=BookieInfoImpl(rack=/rack1, hostname=bk1:3181), bk3:3181=BookieInfoImpl(rack=/rack1, hostname=bk3:3181)}
group2 {bk2:3181=BookieInfoImpl(rack=/rack2, hostname=bk2:3181), bk4:3181=BookieInfoImpl(rack=/rack2, hostname=bk4:3181), bk5:3181=BookieInfoImpl(rack=/rack2, hostname=bk5:3181)}2. 从 bookie affinity group 中删除 bookie。
bin/pulsar-admin bookies delete-bookie-rack -b bk5:31813. 检查是否存在未完全复制数据的 ledger，当我们删除 bookie 之后预计会出现这种情况。
# 在 bk2 节点上执行以下命令
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 bin/bookkeeper shell listunderreplicated4. 停止一个 bookie。
${DOCKER_COMPOSE_HOME}/docker-compose stop bk55. 停用 bookie。
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 bin/bookkeeper shell decommissionbookie -bookieid bk5:31816. 检查已经停用 bookie 中的 ledger。
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 bin/bookkeeper shell listledgers -bookieid bk5:31817. 列出所有的 bookie。
${DOCKER_COMPOSE_HOME}/docker-compose exec bk2 bin/bookkeeper shell listbookies -rw
ReadWrite Bookies :
BookieID:bk2:3181, IP:192.168.48.5, Port:3181, Hostname:bk2
BookieID:bk4:3181, IP:192.168.48.7, Port:3181, Hostname:bk4
BookieID:bk3:3181, IP:192.168.48.6, Port:3181, Hostname:bk3
BookieID:bk1:3181, IP:192.168.48.4, Port:3181, Hostname:bk1相关阅读
引用链接
[1] 《Pulsar Isolation Part IV: Single Cluster Isolation》: https://streamnative.io/blog/engineering/2022-05-26-pulsar-isolation-part-4-single-cluster-isolation/[2] 安装 Docker 环境: https://docs.docker.com/get-docker/
▼关注「Apache Pulsar」，获取干货与动态 ▼
加入 Apache Pulsar 中文交流群
本文分享自微信公众号 - ApachePulsar（ApachePulsar）。
如有侵权，请联系 [email protected] 删除。
本文参与“OSC源创计划”，欢迎正在阅读的你也加入，一起分享。
边栏推荐
- Google Earth Engine APP——一键在线查看全球1984-至今年的影像同时加载一个影像分析
- 在线生成接口文档
- mq消息积压怎么对应
- 八一建军节 | 致敬中国人民解放军
- Go language Go language, understand Go language file operation in one article
- 【CCIG 2022】视觉大模型论坛
- margin 塌陷和重合的理解
- The upgrade of capacity helps the flow of computing power, the acceleration moment of China's digital economy
- 如何封装 svg
- win10 uwp slider 隐藏显示数值
猜你喜欢
网络运维管理从基础到实战-自用笔记（1）构建综合园区网、接入互联网
使用.NET简单实现一个Redis的高性能克隆版（二）
通配符SSL证书不支持多域名吗？
袋鼠云思枢：数驹DTengine，助力企业构建高效的流批一体数据湖计算平台
After EasyCVR is locally connected to the national standard device to map the public network, the local device cannot play and cascade the solution
PHP代码审计8—SSRF 漏洞
Alibaba Cloud International Edition uses ROS to build WordPress tutorial
A group of friends asked for help, but the needs that were not solved in a week were solved in 3 minutes?
ros2订阅esp32发布的电池电压数据
关于使用腾讯云HiFlow场景连接器每天提醒签到打卡
随机推荐
Documentary on Security Reinforcement of Network Range Monitoring System (1)—SSL/TLS Encrypted Transmission of Log Data
＃yyds干货盘点＃ 面试必刷TOP101：链表相加(二)
Win10只读文件夹怎么删除
EasyCVR调用云端录像API接口返回错误且无录像文件生成，是什么原因？
【杰神说说】物联大师2.0版本预告
Regardless of whether you are a public, professional or non-major class, I have been sorting out the learning route for a long time here, and the learning route I have summarized is not yet rolled up
【web自动化测试】playwright安装失败怎么办
”元宇宙“必须具备这些特点
PHP代码审计7—文件上传漏洞
ERC721标准与加密猫
Enterprise survey correlation analysis case
EuROC dataset format and related codes
DHCP&OSPF combined experimental demonstration (Huawei routing and switching equipment configuration)
服务器
部署LVS-DR群集
当前最快的实例分割模型：YOLACT 和 YOLACT++
Nintendo won't launch any new hardware until March 2023, report says
Activity数据库字段说明
ros2订阅esp32发布的电池电压数据
Google Earth Engine APP——一键在线查看全球1984-至今年的影像同时加载一个影像分析