Multi-datacenter operation and detection of concurrent writes

Multi-Data Center Operations

Masterless replication is also suitable for multi-datacenter operations, as it is designed to better tolerate concurrent write conflicts, network outages, and latency spikes.

Cassandra supports cross-datacenter operation in its masterless model in its default configuration: the number of replicas n includes nodes in all datacenters, and in the configuration, you can specify the number of replicas you want to have in each datacenter.Every write from a client is sent to all replicas regardless of the data center, but clients typically only wait for acknowledgments from quorum nodes within their local data center, thus being immune to cross-data center link delays and outages.High-latency writes to other data centers are typically configured to occur asynchronously, although there is some flexibility in configuration [50, 51].

Riak keeps all communication between clients and database nodes local to a data center, so n describes the number of replicas within a data center.Cross-datacenter replication between database clusters happens asynchronously in the background in a style similar to multi-leader replication.

Detect concurrent writes

Dynamo-style DB allows multiple clients to concurrently write to the same K, which may conflict even with strict quorum mechanisms.Similar to multi-master replication, but in DynamoDB, concurrent write conflicts may also occur during read repair or data fetching.

Due to unstable network latency or local failure, requests may arrive in different orders at different nodes.Figure-12 shows that two clients A and B initiate a write request to the primary key X at the same time:

• Node 1 received the write from A, but did not receive the write from B due to node failure
• Node 2 first receives A's write, then B's write
• Node 3 receives B's write first, then A's write

If the node simply overwrites the original K whenever it receives a new write request, the node will be permanently inconsistent, as shown in Figure-12, node 2 considers the final value of X as B, while other nodes consider the value as A.

Replicas should converge to the same value for eventual consensus to be reached.One might hope for automatic handling between replicas, but unfortunately, most implementations are rubbish, and if you don't want to lose data, you have to know a lot about the mechanism of conflict handling within the DB.