When asked during the interview, can redis master-slave copy not answer? These 13 pictures let you understand thoroughly

How to achieve high availability ？ The most important point is redundant data ,redis The redundant storage of data is realized through master-slave replication , So in the Lord redis down After call , Just switch to from , This enables failover , Ensure high availability , Today we are mainly talking about master-slave replication , As for the Lord down After falling , How to switch from , We'll talk about it later .

How to make a backup

I want to see it again redis Before master-slave replication , It is necessary to look at the following three basic concepts .

Backup is divided into Cold standby and Hot standby , If you go deeper, there are How to live .

• Hot standby ： The main database or the main data center undertakes the business flow , At the same time real time Backup data to the slave library or data center . Here's the picture .

• Cold standby ： The main database or main data center undertakes the business flow , Backup data will Timed or offline manual Execute scripts to synchronize data to a slave library or from a data center , Here's the picture .

• Double live ： Two data centers are responsible for the business flow , Data centers are primary and standby to each other , Generally, the primary data center will bear most of the traffic , Another data center will bear a small part of the traffic , Here's the picture .

Be careful ： The definition is directly explained here , As for the failure , How to do manual or automatic failover , This article does not explain , Later on redis When the sentry , I'll talk about it in detail . The above figure only helps us deepen our understanding of the concept of backup .

that redis What kind of backup does it belong to ？ I'm sure you'll understand after reading .

Definition

Master slave copy , It means to put one Redis Server data , Copy to other Redis The server . The former is called the main node (master), The latter is called the slave node (slave), Data replication is one-way , From master to slave only . A master node can have multiple slaves , But a slave node can only have one master node .

Look at the picture ：

We won't talk or explain here redis Installation steps of master and slave , This online blog and official website will have , I believe everyone will configure successfully step by step .

We'll mainly talk about ：

• What has the master-slave replication gone through from the beginning to now , What is synchronized , Data or files ？
• In the master-slave process , If the network is down , What do I do ？
• In the process of synchronization , What factors will increase the pressure of the main reservoir ？

Full amount of synchronization

When we configure master-slave synchronization , Since there has been no synchronization before , Therefore, first, a full amount of data will be synchronized to the slave database .

Master slave connection

After setting the master-slave synchronization configuration , The first step is to establish a connection between the master and the slave , We should get to know each other , Building trust , To start synchronization .

perform slaveof after , What happened? , Look at the picture ：

here Slave From the library will take the initiative and Master Communicate with the main library , send out psync command , The command will carry two parameters to Master, The first parameter is the main library ID（runID）,redis At startup , Will generate a ID, The second is Slave Need from Master Where to start copying data , That is to say Slave Copy Maser Offset of data offset.

Slave For the first time and Master communicate , Because I didn't know at first Master Of ID, So it passed ?
Because it's the first replication , Pass on -1 Indicates full replication for the first time .

next Master received Slave And the corresponding command parameters , At first glance ? and -1 , Then you know this Slave To make a full copy ,Master Will give Slave Send a fullresync command , tell Slave Next, start full replication , And bring your own ID,Slave After receiving these two parameters, they will be saved . Look at the picture ：

send out rdb file

Master Then it will be executed bgsave Generate subprocesses , complete rdb File generation , To generate the rdb After the document , Will send rdb Document to Slave,Slave Will receive rdb file , Before receiving , Will be emptied first. Slave Own database data 【 This process is blocked 】, After emptying , Start receiving rdb file , After receiving , Just load rdb File into memory .

There's another problem here , Is receiving rdb When you file ,Master There may be new write operations , because rdb Is a snapshot of memory at a certain time , So later data , It cannot be transmitted here , here redis A buffer is used to solve , Sending rdb Start , New write request data will be put into this buffer , wait for rdb Once the transmission is complete ,Master Then the data in this buffer will be transferred to Slave,Slave Start receiving , Completion of reception , The master-slave data is consistent , Look at the picture ：

rdb File transfer ：

Buffer transfer ：

Transmission of subsequent commands ：

In fact, the of this buffer is redis called replication buffer ,redis For each connection Master Of Slave Generate such a buffer , Because of every Slave When synchronization starts , It may be different , The synchronization progress must be different , So set up a separate replication buffer.

As long as a Slave and Master Establish a good connection , Corresponding Slave The buffer will be established , If disconnected , Then the buffer will be released .

At the beginning of execution bgsave Generate rdb, All subsequent write requests will be saved to this buffer , That is to say replication buffer in , That is, all subsequent write requests will be sent to Slave.

Look at the picture ：

As shown in the figure above , Every Slave Corresponding to a buffer , That is to say replication buffer.

Incremental replication

In fact, after the whole process above is completed , Full replication is complete , As long as the connection is not broken , Then master-slave replication will continue , So have you ever thought about , If the gateway jitters or breaks , The master-slave connection is disconnected ,redis What will happen ？ Go back to full replication ？

The network is down , What do I do ？

If the network is interrupted , stay redis2.8 I'll go through the full volume generation again before rdb For copy transfer , This is a resource and performance intensive operation .redis2.8 in the future , This process has been optimized , Adopt the mechanism of incremental replication , To reduce the transmission of data , Achieve the purpose of rapid replication , The following mainly explains the process of incremental replication .

Remember when you made a full copy , Will return to Slave An offset ？ Actually Slave After receiving data , This offset will be increased to record the current reception Master How much data . If Master and Slave The offset of is 1000 , Pass on 30 Bytes to Slave, So at this time Master and Slave The offset of should be 1030.

If there's a network outage , Will try again and Master Reconnect the , After the connection , Will send their own offset to Master,Master Will be based on Slave The offset to be sent is determined by Slave Incremental replication or full replication .

Know the general process , So after the network outage , Before resuming the connection , Interrupt the data during this period , It must be out of sync , So where is the data stored ？

Just start master-slave replication , So the new write request is writing replication buffer At the same time , It will also be written to a file called repl_backlog_buffer In the buffer of , This is a ring buffer , Will record Master Receive the offset of the new write request data and the new write command , such Slave After reconnecting , You can then send commands from here to Slave 了 .

look down replication buffer and repl_backlog_buffer（ Ring buffer ） Location map of , Deepen the impression ：

Note that when the connection is not disconnected , These two buffers exist at the same time , If the connection breaks , So it corresponds to Slave Of replication buffer The buffer will be deleted .

In fact, each segment of the ring records the current data and offset , With the currently written offset Growing , Because this is a ring buffer , It will happen Overlay the previous data .

Ring buffer ,repl_backlog_buffer The record is the current Master Cumulative number of new write requests received offset value （master_repl_offset）, Said is Master Progress , When a network outage occurs , be-all Slave Will be with Master Of offset Compare , So it's all Slave Public .

Incremental replication process

• Slave Try sending psync close Master Of runID and Their own offset （slave_repl_offset） .

• Master Received psync after , Determine whether to perform incremental replication or full replication .

Incremental replication is possible , Look at the picture ：

If received runID and Master runID identical , meanwhile repl_backlog_buffer Buffer offset Will be with Slave Sending a offset Compare , If the gap between the master and slave nodes does not exceed the length of the ring buffer , Or no ferrule , That is, the data before overwriting will not occur , that Master Will reply Continue to Slave, tell Slave Incremental replication is available .

If you find that runID And now, Master atypism , perhaps Master and slave offset The gap is more than repl_backlog_buffer Length of buffer , Then it will be copied in full , Not much here .

• Master send out offset After the order to Slave, Look at the picture .

because Slave Sent by offset yes 998 , Now? Master Of offset yes 1000, therefore Master Will be able to 998-1000 The command between continues to pass to Slave, In this way, incremental transmission is achieved .

summary

Up to now the whole redis The process of master-slave replication is explained , Now let's make a summary .

There are two types of master-slave synchronization ：

• Full amount of synchronization

Full amount of synchronization redis Will execute bgsave To generate rdb file , And then send it to the slave Library , Before receiving from the library, the data from the library will be emptied , Prevent data pollution caused by previous data , After receiving rdb After the document , Will load rdb File to memory , This is a synchronization, but it's not finished , Generating rdb When you file , There will be new write requests , At this point, these write requests are cached in a buffer , This buffer is called replication buffer, After loading from the library rdb after , Will receive all write commands to this buffer , Now the full copy is over .

Because of production rdb It will block the main thread , This process is resource intensive , If one master and multiple slaves are adopted , Then it is bound to increase the pressure on the main reservoir , You can choose one from , Then split one or more slaves from the library , To reduce the pressure on the main reservoir .

If you want to generate quickly rdb file , It should be reduced redis Set the memory size , This produces rdb The file will be soon , Reduce blocking time .

• The incremental synchronization

If the master-slave is disconnected ,redis The master database will judge whether to perform full replication or incremental replication , The main database will be sent from the database runID And copy from library offset, If runID And the main warehouse ID identical , And master-slave offset The gap does not exceed repl_backlog_buffer Length of buffer , Will copy offset Between repl_backlog_buffer The order to Slave.

Two buffers ：

• replication buffer

replication buffer After the master and slave libraries are successfully created , After the master-slave is disconnected , This buffer will also be deleted by the main library , Transfer of copy commands between master and slave libraries , They're going through this buffer, And this buffer Is unique to each slave Library .

• repl_backlog_buffer

Before starting command transmission , Will set up this buffer, This buffer Record the current Master New write command received offset And the command itself , It's all Slave Public buffer,Slave send out psync after , Hui He Master Of offset Compare , To decide whether to make incremental replication .

Be careful ：

1、redis The memory size of the instance should not be set too large , This can shorten the generation time rdb Time of file , At the same time, it can also shorten the time of full replication , Reduce bandwidth usage .
2、 If you disconnect from the library and the main library, the timeout is very long , that repl_backlog_buffer The data in the buffer is likely to be overwritten , Then it will degenerate into full replication , Now you can set repl_backlog_size Set this parameter larger .
3、replication buffer, This buffer should also pay attention to , If receiving from the library is slow , This buffer will be full ,redis Maybe OOM 了 , If this buffer Full of redis What will happen to ,redis Provides
client-output-buffer-limit Parameters limit this buffer Size , If the full , The master library will be disconnected from the slave , Delete buffer, If you ask for a link again , Could create a vicious circle .

---

Today's sharing is here , It's not easy to code , Looking forward to your give the thumbs-up 、 Focus on 、 forward , thank you .