2022-01-27 redis cluster brain crack problem analysis

Catalog

Abstract :

cluster Cluster fault detection and master-slave switching :

redis-cluster The cause of the brain crack problem :

Brain fissure resolution strategy :

Options :

---

Abstract :

analysis redis-cluster Cluster pattern , Strategies to solve the problem of cerebral fissure

cluster Cluster fault detection and master-slave switching :

NDB.Redis- Data security -cluster colony

redis-cluster The cause of the brain crack problem :

Set the cluster to have cluster1, Including one master and one slave , master1 and slave1. client And master1 Connect .

At this time, there are two network partitions :

1. client And master1 Network partition
2. cluster The rest of the cluster master Node and master1 Network partition

Preconditions for the occurrence of cerebral fissure ：

1. cluster More than half of the cluster master node , And master1 Between nodes , Network disconnection occurs
2. client And master1 Your network connection is normal

cluster More than half of the cluster master node , And master1 Between nodes , Network disconnection occurs :

Lead to :

1. cluster Cluster decision master1 Get into FAIL state
2. slave1 Received master1 Get into FAIL Broadcast of status , Start applying for master-slave switching voting
3. slave1 By more than half of master The node voted , slave1 Begin to enter master state , And broadcast new master
4. To the whole cluser For clusters , The original master1 It's offline , master The nodes are made up of slave1 Switch complete
5. about cluster1 Come on , Now the latest data , It's from the original slave1 Provide
6. For the original mastee1
   1. The original master1 The data of the node will be lost
   1. If cluster The cluster decides to go online again , Is judged to be slave state , From the original slave1 After the master-slave switch , Make a full copy slave1 The data of

client And master1 Your network connection is normal :

here client There are the following conditions ：

1. client Don't ask again cluster Cluster routing information in the cluster
2. client And master1 The network channel of is normal

Lead to ：

1. client The data of continues to be written master1
2. But for cluster colony , Determined master1 Get into FAIL state , And has elected a new master node
3. Lead to client write in master1 Your data will be lost

Brain fissure resolution strategy :

A strategy : client Heartbeat detection update cluster Cluster information , Reduce and judge as FAIL Of master1 The connection time of

specific working means :

1. client The role of is represented by the cluster proxy be responsible for
2. To shorten the proxy Node query cluster Time period of routing table status
   1. slave node , and proxy Of the query cluster node , Received at the same time master1 Of FAIL State change
   1. slave Time of master-slave switching , Include
   1. cluster One in the cluster mater determine master1 by FAIL, Then broadcast the message , It is considered that all nodes receive master1 Get into FAIL The state of
   2. You need to proxy The time period of heartbeat detection < Time of master-slave switching

• • • 1. To all the master Send a vote request - Time consuming :
      2. Accept more than half of master The vote of the corresponding - Time consuming :
      3. It's up to you to switch - Time consuming :
    • proxy The time to query the routing table from the static node
      1. ClusterServerPool::RefreshInterval Configure query cycle , It can be accurate to us, Microsecond

Strategy two : master1 The node determines whether it has been connected with cluster Cluster network isolation , If isolated , Write is rejected

principle : master Judge whether you have not exchanged data with the slave node , If you haven't done data exchange for a long time , It is considered that this node has been cluster The cluster decides to go offline , Then this master The node refused to write

redis.conf Configuration properties of ：

1. min-replicas-to-write: master Nodes at least slave Number of nodes , otherwise master Write denied
2. min-replicas-max-lag: master Nodes are at least related to slave Node data heartbeat ping Time , Beyond that time , be master Write denied

Use min-replicas-max-lag Configuration feasibility analysis :

explain : This configuration is made up of master Judgment and slave Copy time , To decide whether you can be written , So as to avoid being judged by the cluster FAIL when , Continue to the old master write in

The problem is :

1. master Of FAIL State determination , And master The rejection of the node is written , Are different judgment paths
   1. master1 Get into FAIL, By cluster Others in the cluster master The node is responsible for determining
   2. master1 The node refused to write , It is master Node judgment and slave Node ping interval
2. cluster When one master and one slave are deployed , Master slave switchover occurs when the master node fails , The new main node will refuse to write
   1. master Downtime ,slave Switch to a master, Now the new master No, slave
   2. Because no slave, Lead to min-replicas-max-lag The decision timed out , Lead to a new master Write denied , This leads to cluster Fragmentation is not available

Options :

1. Client pass proxy When the connection , To shorten the proxy The period of querying the status of the cluster
2. Client direct connection cluster Of master Node time , temporary (630 edition ) Don't deal with it