Pulsar geo replication/ disaster recovery / regional replication

Catalog

1. Replication mechanism /mechanisms

（1）Pulsar Asynchronous geo replication in

（2） adopt BookKeeper Synchronous geo replication

2. Copying model /patterns

（1） Mesh topology replication

（2）Active-active Copy

（3）Active-standby replication / Take the initiative - Standby replication ​

（4） Aggregate replication / Aggregation replication

---

Pulsar Geo replication mechanism is usually used for disaster recovery , Data that supports persistent storage is replicated across multiple data centers . for example , Your app is publishing data in a region , And you want to process this data for consumption in other regions . utilize Pulsar Geographical replication mechanism , Messages can be generated and consumed in different geographic locations .

The following figure illustrates the process of geographic replication . Whenever the producer （P1、P2 and P3） In three different clusters T1 When the news is released , At this point, the message is immediately replicated in the cluster . Once the message is copied , consumer （C1 and C2） These messages can be consumed in their respective clusters .

1. Replication mechanism /mechanisms

Geographic replication mechanisms can be divided into synchronous and asynchronous geographic replication mechanisms .Pulsar Both replication mechanisms are supported .

（1）Pulsar Asynchronous geo replication in

An asynchronous geo replication cluster consists of multiple physical clusters set up in different data centers .Pulsar The messages generated by the topic are first persisted to the local cluster , Then from brokers Asynchronous replication to remote cluster .

Under normal circumstances , When there is no connection problem , The message will be copied immediately , Send to local consumers at the same time . Usually , The end-to-end delivery delay is determined by the network round trip time between data centers （RTT） Definition . Applications can create producers and consumers in any cluster , Even if the remote cluster cannot access （ for example , During network partitioning ）.

Asynchronous geo replication provides low latency , Because some data has not been copied , There may be replication delays , Lead to The data consistency guarantee is weak .

（2） adopt BookKeeper Synchronous geo replication

In synchronous geo replication , Synchronous replication of data to multiple data centers , The client must wait for confirmation from other data centers . As shown in the figure below , When a client sends a write request to a cluster , The written data will be copied to the other two data centers . Only when most data centers （ In this case , There are at least 2 Data centers ） Confirmed Write is persisted when , To confirm the write request to the client .

Pulsar Synchronous geo replication in is performed by BookKeeper Realization . A synchronous geo replication cluster consists of a bookies Cluster and a brokers colony , And an overall situation Zookeeper install （Zookeeper Integrate and operate in multiple data centers ）. You need to configure a BookKeeper region-aware Placement strategy , To store data across multiple data centers , And ensure the availability constraints of write operations （availability constraints on writes）.

Synchronous geo replication provides the highest availability , It also ensures stronger data consistency between different data centers . however , Your application must Bear the additional cost of cross data center latency .

2. Copying model /patterns

Pulsar Provides great flexibility for customizing replication policies . You can set different replication modes , Provide replication strategies for applications between multiple data centers .

（1） Mesh topology replication

Use mesh topology replication （Full-mesh replication） And apply selective message replication , You can customize replication policies and topologies between any number of data centers .

（2）Active-active Copy

Active-active Replication is a variant of mesh topology replication , There are only two data centers . Producers can run and generate messages in any data center , Consumers can consume all messages from all data centers .

（3）Active-standby replication / Take the initiative - Standby replication ​

Active-standby Copy is Active-active A variant of replication . The producer sends a message to the active data center , At the same time, copy the message to the standby data center for backup . If the active data center shuts down , The standby data center will take over and become the active data center .

（4） Aggregate replication / Aggregation replication

The aggregate replication pattern is often used to pull messages from the edge （ The brink of disaster ） Copy to cloud . for example , Suppose you are in 3 There are... In front data centers 3 Clusters , There is an aggregation cluster in a central data center , And you want to replicate messages from multiple front-end data centers to the central data center for aggregation . then , You can create a separate namespace for topics used by each front-end data center , And assign the aggregated data center to these namespaces .