Master slave replication of redis

Redis Master-slave replication

1. Internet “ Three tenors ” framework

1. High performance
2. High availability
3. High concurrency

2. High availability

stand-alone redis The risks and problems of
problem 1. Machine fault
The phenomenon ： Hard disk failure 、 System crash
The essence ： Data loss , It's likely to be a catastrophic blow to the business
Conclusion ： Basically, it will be abandoned redis.
problem 2. Capacity of the bottleneck
The phenomenon ： Out of memory , from 16G Upgrade to 64G, from 64G Upgrade to 128G, Unlimited upgrade memory
The essence ： poor , Hardware conditions can't keep up with
Conclusion ： Abandon use redis
Conclusion ：
To avoid a single point Redis Server failure , Prepare multiple servers , Connected to each other . Keep multiple copies of your data on different servers , come together , And keep the data synchronized . Even if one of the servers goes down , Other servers can still provide services , Realization Redis High availability , At the same time, realize redundant data backup .

3. Introduction to master-slave replication

Data provider master, master server , Master node , Main library , Primary client
Receiving party ：slave, From the server , From the node , Slave Library , From the client
Problems to be solved ： Data synchronization
The core work ：master Copy data from to slave in

Master slave replication is about to master Data in real time 、 Effectively copy to slave in
features ： One master You can have multiple slave, One slave Only one master
duty ：
master:
1. Writing data
2. When writing , Automatically synchronize the changed data to slave
3. Reading data （ Negligible ）
slave:
1. Reading data
2. Writing data （ prohibit ）

4. High availability cluster

5. Master-slave replication

Master-slave replication
1. Read / write separation ：master Write 、slave read , Improve the load capacity of the server
2. Load balancing ： Based on the master-slave structure , Cooperate with the separation of reading and writing , from slave Share responsibility master load , And according to the change of demand , change slave The number of , Sharing the load of data reading through multiple slave nodes , Greatly improved Redis Server concurrency and data throughput
3. Fault recovery ： When master When something goes wrong , from slave Provide services , Fast fault recovery
4. data redundancy ： Hot data backup , It's a way of data redundancy beyond persistence
5. High availability cornerstone ： Based on master-slave replication , Construction of sentry mode and cluster , Realization Redis High availability solution for

6. Master slave copy workflow

The master-slave replication process can be roughly divided into 3 Stages :
1. Establish connection phase （ Preparation stage ）:
establish slave To master The connection of , send master Able to identify slave, And save slave Port number
2. Data synchronization phase
3. Command propagation stage

7. Stage 1 establish connection stage

7.1 Establish the connection process

Establish connection phase workflow
step 1： Set up master The address and port of , preservation master Information
step 2： establish socket Connect
step 3： send out ping command （ Timer task ）
step 4： Authentication
step 5： send out slave Port information
thus , Master slave connection successful ！

state ：
slave：
preservation master Address and port of
master：
preservation slave The port of
The overall ：
That creates a connection between socket

slave system information
1.master_link_down_since_seconds
2.masterhost
3.masterport
master system information
1.slave_listening_port( Multiple )

7.2 Establish connection mode

Mode one ： Client sends command

slaveof <masterip> <masterport>

Mode two ： Start server parameters

redis-server -slaveof <masterip> <masterport>

Mode three ： Server configuration :
slaveof

7.3 Authorized access

If master Set a password , You can modify the following parameters in the configuration file :
masterauth

7.4 disconnect

Client sends command :
notes ：slave After disconnection , Existing data will not be deleted , Just don't accept it anymore master Data sent

slaveof no one

8. demonstration ( Stage 1 establish connection stage )

Open four windows ,6379 The server is used to start the host to view logs 、6380 The server is used to start the slave to view the log 、master For client operation 、salve For client operation

6379 The server :
First, you need to modify two configuration files , send redis The front desk starts , The log is no longer printed into the log file , Instead, print to the console ; Turn off the daemon , Also close the log file

Connection mode one

6379 The server 、6380 The server
start-up 6379 Port of redis、 start-up 6380 Port of redis

salve 、6379 The server 、6380 The server :
6380 As slave, Connect 6379.6379 The server 、6380 The server has log printing

master、slave：
master to set A value ,slave Sure get To

Connection mode 2

6380 The server ：
First ctrl +C close 6380 The server , then 6380 Take it with you when the server starts slave Connect

master、slave：

Connection mode three

ctrl+c close 6380 The server

6380 The server ：
modify 6380 The configuration file

6380 The server ：
start-up 6380 Port of redis

View status information

master：

salve：

9. Stage 2 data synchronization stage

9.1 Data synchronization process

stay slave First connection master after , Copy master All data in to slave, take slave The database status of is updated to master Current database state .
Data synchronization phase workflow
step 1： Request to synchronize data
step 2： establish RDB Synchronous data
step 3： recovery RDB Synchronous data
step 4： Request partial synchronization data
step 5： Restore partially synchronized data
thus , Data synchronization is complete ！

state ：
slave：
have master All the data at the end , contain RDB Data received by the process
master：
preservation slave The location of the current data synchronization
The overall ：
Data cloning is completed between

9.2 Data synchronization master explain

Data synchronization phase master explain :
1. If master The amount of data is huge , Data synchronization phase should avoid the peak traffic , Avoid creating master Blocking , Affect the normal implementation of business
2. The copy buffer size is not set properly , Can cause data overflow . If the full copy cycle is too long , It is found that the data has been lost during partial replication , A second full copy is required , the slave In a dead cycle , Parameters ：
repl-backlog-size 1mb （1mb yes 1 mega ）
3.master The proportion of single machine memory occupying host memory should not be too large , It is recommended to use 50%-70% Of memory , leave 30%-50% Is used to execute bgsave Command and create a copy buffer

9.3 Data synchronization phase slave explain

1. To avoid slave Make full copies 、 Server response blocked or data out of sync during partial replication , It is suggested to close the external service during this period , Parameters ：
slave-serve-stale-data yes|no
2. Data synchronization phase ,master Send to slave The information is understandable master yes slave A client of , Active direction slave dispatch orders
3. Multiple slave At the same time master Request data synchronization ,master Sent RDB More documents , It's going to have a huge impact on bandwidth , If master Not enough bandwidth , Therefore, data synchronization needs to be based on business requirements , Right amount of peak shifting
4.slave Too much time , It is suggested to adjust the topology structure , From one master to many, from the structure to the tree structure , The node in the middle is master, It's also slave. Pay attention to the tree structure , Because of the depth of the hierarchy , That leads to the deeper slave And the top master The data synchronization delay between is large , Data consistency is getting worse , We should choose carefully

10. Stage 3 Command propagation stage

10.1 Command propagation process

When master After the database state is modified , The database status of the master-slave server is inconsistent , At this point, you need to synchronize master-slave data to a consistent state , Synchronous action is called command propagation .master Send the received data change command to slave,slave After receiving the command, execute the command

10.2 Partial replication in the command propagation phase

In the stage of command propagation, the network is broken :
1. The network is broken and connected , The solution is “ Can be ignored ”
2. Short time network outage , The solution is “ Partial reproduction ”
3. Long time network interruption , The solution is “ Copy in full ”

Three core elements of partial replication
1. Running the server id（run id）
2. Replication backlog buffer for primary server
3. The replication offset of the master and slave servers

10.3 Running the server id

1. Concept ： Server running ID It's the ID of each server running every time , Multiple runs can be generated from a single server id
2. form ： function id from 40 Bit characters make up , It's a random hexadecimal character
for example ：fdc9ff13b9bbaab28db42b3d50f852bb5e3fcdce
3. effect ： function id Used to transfer between servers , Identification
If you want to do both operations on the same server , Each operation must carry the corresponding run id, Used to identify the other side
4. Realization way ： function id Generated automatically at the start of each server ,master In the first connection slave when , Will run their own ID Send to slave,slave Save this ID
notes ： adopt info Server command , You can check your own runid

info Server

master:

slave:

10.4 Copy buffer

Concept ： Copy buffer , Also known as copy backlog buffer , It's a first in, first out （FIFO） Queues , Used to store commands executed by the server , Every time you pass a command ,master Will record the command spread , And stored in the copy buffer

10.5 How copy buffer works

1. form ：
1) Offset
2) Byte value
2. working principle ：
1) adopt offset Distinguish between different slave Differences in current data dissemination
2)master Record the information that has been sent offset
3)slave Record the received information corresponding to offset
3. Concept ： Copy buffer , Also known as copy backlog buffer , It's a first in, first out （FIFO） Queues , Used to store commands executed by the server , Every time you pass a command ,master Will record the command spread , And stored in the copy buffer . The default data storage size of the copy buffer is 1M, Because the size of storage space is fixed , When the number of queued elements is greater than the queue length , The first element to join the team will be ejected , And new elements are put into the queue
3. origin ： When each server starts , If there is AOF Or connected to become master node , That is, create a copy buffer
4. effect ： Used to hold master All instructions received （ Instructions that only affect data changes , for example set,select）
5. Data sources ： When master When receiving instructions from the master client , In addition to executing instructions , The instruction is stored in the buffer
notes ：\r \n It's return 、 Line break means

10.6 Offset offset

1. Concept ： A number , Describes the instruction byte position in the copy buffer
2. classification ：
1)master Copy offset ： Records are sent to all slave The position of the instruction byte of （ Multiple ）
2)slave Copy offset ： Record slave receive master The position of the instruction byte sent （ One ）
3. Data sources ：
1)master End ： Send once, record once
2)slave End ： Receive once, record once
4. effect ： Sync information , comparison master And slave The difference of , When slave After the break , Recover data usage

11. Data synchronization + Command propagation phase workflow

12. heartbeat

12.1 heartbeat

Enter the command propagation stage ,master And slave We need to exchange information between them , Use heartbeat mechanism for maintenance , Keep both sides connected online
master Heartbeat task ：
1. Instructions ：PING
2. cycle ： from repl-ping-slave-period decision , Default 10 second
3. effect ： Judge slave Whether online
4. Inquire about ： obtain slave Last connection interval ,lag It's maintained at 0 or 1 As normal

INFO replication

slave Heartbeat task
1. Instructions ：REPLCONF ACK {offset}
2. cycle ：1 second
3. effect 1： report slave Own copy offset , Get the latest data change instructions
4. effect 2： Judge master Whether online

master：

12.2 Notes during the heartbeat phase

1. When slave Most of them are offline , Or when the delay is too high ,master To ensure data stability , All information synchronization operations will be rejected ,slave Less quantity 2 individual , Or all slave All delays are greater than or equal to 10 seconds , closed master Write function , Stop data synchronization , The parameters are as follows ：
min-slaves-to-write 2
min-slaves-max-lag 8
2.slave The quantity is from slave send out REPLCONF ACK Order confirmation
3.slave Delay by slave send out REPLCONF ACK Order confirmation

13. Master slave copy complete workflow

14. Master slave replication FAQ

14.1 Frequent full replication

With the operation of the system ,master The amount of data will be larger and larger , once master restart ,runid It's going to change , It will lead to all slave Full copy operation of
Internal optimization and adjustment plan ：
1.master Create inside master_replid Variable , Use runid The same policy generation , length 41 position , And send it to all slave
2. stay master Execute command when closing shutdown save, Conduct RDB Persistence , take runid And offset Save to RDB In file , namely repl-id repl-offset, adopt redis-check-rdb Command to view the information
3.master Load after restart RDB file , Restore data . After restart , take RDB In the file repl-id And repl-offset Load into memory , adopt info Command to view the information
master_repl_id = repl master_repl_offset = repl-offset
3. effect ：
Save last time locally runid, Restore the value after restart , Make all slave I think it's the same as before master

redis-check-rdb

master:

14.2 Frequent full replication

1. Problem phenomenon :
1) The network environment is not good , There's a network outage ,slave No service
2. Question why :
Copy buffer too small , After the disconnection slave Of offset Transboundary , Trigger full replication
3. final result :
slave Repeat full replication
4. Solution :
Modify the copy buffer size , The parameters are as follows ：
repl-backlog-size
5. The recommended settings are as follows ：
1) Calculation from master To slave The average length of reconnection second
2) obtain master The average amount of write command data generated per second write_size_per_second
3) Optimal copy buffer space = 2 * second * write_size_per_second

14.3 Network interruption

1. Problem phenomenon :
master Of CPU Occupy too much or slave Frequent disconnection
2. Question why :
1)slave Every time 1 Seconds to send REPLCONF ACK Order to master
2) When slave When you get a slow query （keys * ,hgetall etc. ）, It's going to take up a lot of CPU performance
3)master Every time 1 Second call copy timer function replicationCron(), comparison slave I found that I didn't respond for a long time
3. final result :
master Various resources （ Output buffer 、 bandwidth 、 Connections etc. ） Seriously occupied
4. Solution :
1) By setting a reasonable timeout , Confirm whether to release slave, This parameter defines the threshold for the timeout （ Default 60 second ）, Beyond this value , Release slave, The parameters are as follows ：
repl-timeout

14.4 Network interruption

1. Problem phenomenon :
slave And master Connection is broken
2. Question why :
1)master send out ping Command frequency is low
2)master Set a shorter timeout
3)ping Instruction packet loss exists in the network
3. Solution :
Improve ping How often instructions are sent , Timeout time repl-time At least ping Command frequency 5 To 10 times , otherwise slave It's easy to determine the timeout , The parameters are as follows ：
repl-ping-slave-period

14.5 Data inconsistency

1. Problem phenomenon :
Multiple slave Getting the same data out of sync
2. Question why :
Network information is not synchronized , There's a delay in data transmission
3. Solution :
1) Optimize the network environment between master and slave , They are usually deployed in the same computer room , Pay attention to this phenomenon when using Alibaba cloud and other cloud servers
2) Monitor the master-slave delay （ adopt offset） Judge , If slave Too much delay , Temporarily block the program for the slave Data access
3)slave-serve-stale-data yes|no
slave-serve-stale-data Parameter set to yes, Master slave replication , The slave server can respond to client requests , Only respond to info、slaveof Wait for a few orders （ Use with caution , Unless there is a high demand for data consistency ）;
slave-serve-stale-data Parameter set to no, Master slave replication , The slave server will block all requests , When there is a client request, return “SYNC with master in progress”;