Interviewer: how to deal with the data loss of redis master-slave cluster switching?

One 、 Data loss

• Asynchronous replication lost synchronization
• The cluster produces brain fissure and data is lost

1. Asynchronous replication is missing

about Redis Data replication between master and slave nodes , Asynchronous replication , When the client sends a write request to master Node time , The client will return OK, And then synchronize to each slave In nodes .

If at this time master Haven't had time to sync to slave Node downtime , that master Data in memory will be lost ;

If master Is it possible to ensure that the persistence setting data is not lost when it is enabled in the ？ The answer is No . stay master After downtime ,sentinel Cluster detected master failure , Re elect a new master, If the old master Restart after failure recovery , Then it needs to synchronize the new master The data of , Now the new master Is empty （ Suppose no data is written during this time ）. So old master The data in will be refreshed , At this time, the data will still be lost .

2. Clusters produce brain fissures

First, we need to understand the brain cleft phenomenon of clusters , It's like a person has two brains , So who is in control ？ In a distributed cluster , Distributed collaboration framework zookeeper It solved the problem well , Solve the problem by controlling more than half of the machines .

So in Redis in , What is the phenomenon of data loss caused by cluster brain fissure ？

Suppose we have a redis colony , Under normal circumstances client Will send to master Send a request , And then synchronize to salve,sentinel Monitoring the cluster , Automatic failover in case of cluster failure .

here , For some reason , For example, network reasons , There are partitions in the cluster ,master And slave The nodes are disconnected ,sentinel I haven't been in touch for a while and think master fault , And then re elect , take slave Switch to new master.

however master There may be no failure , It's just that the network generates partitions , here client Still in the old master Write the data on , And new master No data in , If the problem is not found and handled in time, it may be old master Pile up a lot of data in . After discovering the problem , old master Reduced to slave Synchronize NEW master data , Then the previous data is refreshed , Massive data loss .

After understanding the above two data loss scenarios , How can we ensure that data is not lost ？ In distributed systems , Measure the availability of a system , We usually say 4 individual 9,5 individual 9 The system has reached high availability （99.99%,99.999%, It is said that Taobao is 5 individual 9）. about redis colony , We cannot guarantee that the data will not be lost at all , We can only make as little data loss as possible .

Two 、 How to ensure as little data loss as possible ？

stay redis There are two parameters in the configuration file that we can set ：

min-slaves-to-write 1min-slaves-max-lag 10

min-slaves-to-write By default 0,min-slaves-max-lag By default 10.

Take the above configuration as an example , These two parameters represent at least 1 individual salve Of and master The synchronous replication delay for cannot exceed 10s, Once all of slave Replication and synchronization latency reached 10s, So at this time master Will not accept any requests .

We can reduce min-slaves-max-lag The value of the parameter , In this way, a large amount of data loss can be avoided in case of failure , Once it is found that the delay exceeds this value, it will not go to master Middle write data .

So for client, We can take measures to downgrade , Write data to local cache and disk temporarily , Write back... After a period of time master To ensure that data is not lost ; You can also write data to kafka Message queue , Spend... Every other time kafka Data in .

Through the setting of the above two parameters, we can reduce the loss of data as much as possible , Specific values also need to be set in a specific environment .