Redis (VI) - master-slave replication

Redis（ 6、 ... and ）—— Master slave copy

1、 brief introduction

Multi server connection scheme

• Data provider ：master
  • master server , Master node , Main library
  • Primary client
• The party receiving the data ：slave
  • From the server , From the node , Slave Library
  • From the client
• Problems to be solved
  • Data synchronization
• The core work
  • master The data of Copy To slave in

Master slave copy

Master slave replication is about to master Data in real time 、 Effective Copy To slave in

features ： One master You can have multiple slave, One slave Only one master

duty ：

• master:
  • Writing data
  • When writing , Automatically change the changed data Sync To slave
  • Reading data （ Negligible ）
• slave:
  • Reading data
  • Writing data （ prohibit ）

2、 effect

• Read / write separation ：master Write 、slave read , Improve the load capacity of the server
• Load balancing ： Based on the master-slave structure , Cooperate with the separation of reading and writing , from slave Share responsibility master load , And according to the change of demand , change slave The number of , Sharing the load of data reading through multiple slave nodes , Greatly improved Redis Server concurrency and data throughput
• Fault recovery ： When master When something goes wrong , from slave Provide services , Fast fault recovery
• data redundancy ： Hot data backup , It's a way of data redundancy beyond persistence
• High availability cornerstone ： Based on master-slave replication , Construction of sentry mode and cluster , Realization Redis High availability solution for

3、 Workflow

executive summary

• The master-slave replication process can be roughly divided into 3 Stages
  • Establish connection phase （ Preparation stage ）
  • Data synchronization phase
  • Command propagation stage

[

Stage 1 ： Establishing a connection

• establish slave To master The connection of , send master Able to identify slave, And save slave Port number

  
  Master slave connection （slave Connect master）

• Mode one ： Client sends command

  slaveof <masterip> <masterport>
  

• Mode two ： Start server parameters

  redis-server -slaveof <masterip> <masterport>
  

• Mode three ： Server configuration （ Commonly used ）

  slaveof <masterip> <masterport>
  

  

Master slave disconnect

• client dispatch orders

  slaveof no one
  

  • explain ： slave After disconnection , Existing data will not be deleted , Just don't accept it anymore master Data sent

Authorized access

• master The client sends a command to set the password

  requirepass <password>
  

• master Profile settings password

  config set requirepass <password> 
  config get requirepass
  

• slave The client sends a command to set the password

  auth <password>
  

• slave Profile settings password

  masterauth <password>
  

• slave Start the server and set the password

  redis-server –a <password>
  

Stage two ： Data synchronization phase

• Copy in full

  • take master perform bgsave Before ,master All data in is synchronized to slave in

• Partial reproduction

  （ Incremental replication ）

  • take master perform bgsave In operation , Newly added data （ Copy the data in the buffer ） Pass to slave,slave adopt bgrewriteaof Command to recover data

Data synchronization phase master explain

1. If master The amount of data is huge , The data synchronization phase should Avoid peak traffic , avoid cause master Blocking , Affect the normal implementation of business
2. The copy buffer size is not set properly , Can cause data overflow . If the full copy cycle is too long , It is found that the data has been lost during partial replication , A second full copy is required , the slave fall into Dead cycle state .

repl-backlog-size 1mb

1. master The proportion of single machine memory occupying host memory should not be too large , It is recommended to use 50%-70% Of memory , leave 30%-50% Memory is used to perform That's ok bgsave Command and create a copy buffer

Data synchronization phase slave explain

1. To avoid slave Make full copies 、 Server response blocked or data out of sync during partial replication , Proposed closure External services during this period

slave-serve-stale-data yes|no

1. Data synchronization phase ,master Send to slave The information is understandable master yes slave A client of , Active direction slave dispatch orders
2. Multiple slave At the same time master Request data synchronization ,master Sent RDB More documents , It's going to have a huge impact on bandwidth , If master Not enough bandwidth , Therefore, data synchronization needs to be based on business requirements , Right amount of peak shifting
3. slave Too much time , It is suggested to adjust the topology structure , From one master to many, from the structure to the tree structure , The node in the middle is master, It's also slave. Pay attention to the tree structure , Because of the depth of the hierarchy , That leads to the deeper slave And the top master The data synchronization delay between is large , Data consistency is getting worse , We should choose carefully

Stage three ： Command propagation stage

• When master After the database state is modified , The database status of the master-slave server is inconsistent , At this point, you need to synchronize master-slave data to a consistent state , Sync This action is called Command propagation

• master Send the received data change command to slave,slave After receiving the command, execute the command

• The master-slave replication process can be roughly divided into 3 Stages

  • Establish connection phase （ Preparation stage ）
  • Data synchronization phase
  • Command propagation stage

Partial replication in the command propagation phase

• In the stage of command propagation, the network is broken

  • The network is broken and connected
  • Short time network outage
  • Long time network interruption

• Partially copied Three core elements

  • Running the server id（run id）
  • Replication backlog buffer for primary server
  • The replication offset of the master and slave servers

Server running ID（runid）

• Concept ： Server running ID It's the ID of each server running every time , Multiple runs can be generated from a single server id
• form ： function id from 40 Bit characters make up , It's a random hexadecimal character for example - -
  • fdc9ff13b9bbaab28db42b3d50f852bb5e3fcdce
• effect ： function id Used to transfer between servers , Identification
  • If you want to do both operations on the same server , Each operation must carry the corresponding run id, Used to identify the other side
• Realization way ： function id Generated automatically at the start of each server ,master In the first connection slave when , Will run their own ID Send to slave,slave Save this ID, adopt info Server command , You can view the runid

Copy buffer

• Concept ： Copy buffer , Also known as copy backlog buffer , It's a fifo （FIFO） Queues , Used to store the life executed by the server Make , Every time you pass a command ,master Will record the command spread , And stored in the copy buffer
• origin ： When each server starts , If there is AOF Or connected to become master node , That is, create a copy buffer
• effect ： Used to hold master All instructions received （ Instructions that only affect data changes , for example set,select）
• Data sources ： When master When receiving instructions from the master client , In addition to executing instructions , The instruction is stored in the buffer

How replication buffers work inside

• form

  • Offset
  • Byte value

• working principle

  • adopt offset Distinguish between different slave Differences in current data dissemination
  • master Record Has been sent The information corresponding to offset
  • slave Record Received The information corresponding to offset

Master slave copy offset （offset）

• Concept ： A number , Describes the instruction byte position in the copy buffer
• classification ：
  • master Copy offset ： Records are sent to all slave The position of the instruction byte of （ Multiple ）
  • slave Copy offset ： Record slave receive master The position of the instruction byte sent （ One ）
• Data sources ： master End ： Send once, record once slave End ： Receive once, record once
• effect ： Sync information , comparison master And slave The difference of , When slave After the break , Recover data usage

Data synchronization + Command propagation phase workflow

heartbeat

• Get into Command propagation phase ,master And slave We need to exchange information between them , Use heartbeat mechanism for maintenance , Keep both sides connected online
• master heartbeat ：
  • Instructions ：PING
  • cycle ： from repl-ping-slave-period decision , Default 10 second
  • effect ： Judge slave Whether online
  • Inquire about ：INFO replication obtain slave Last connection interval ,lag It's maintained at 0 or 1 As normal
• slave Heartbeat task
  • Instructions ：REPLCONF ACK {offset}
  • cycle ：1 second
  • effect 1： report slave Own copy offset , Get the latest data change instructions
  • effect 2： Judge master Whether online

Notes during the heartbeat phase

• When slave Most of them are offline , Or when the delay is too high ,master To ensure data stability , All information synchronization operations will be rejected

  min-slaves-to-write 2 
  min-slaves-max-lag 8
  

  • slave Less quantity 2 individual , Or all slave All delays are greater than or equal to 10 seconds , closed master Write function , Stop data synchronization

• slave The quantity is from slave send out REPLCONF ACK Order confirmation

• slave Delay by slave send out REPLCONF ACK Order confirmation

Complete process

common problem

Frequent network interruptions

Data inconsistency