当前位置：网站首页>Redis master-slave replication cluster and recovery ideas for abnormal data loss ＃ yyds dry goods inventory ＃

Redis master-slave replication cluster and recovery ideas for abnormal data loss ＃ yyds dry goods inventory ＃

2022-07-05 01:04:00 【Jiangxiaolong's technology blog】

Redis Master slave replication cluster and recovery of abnormal data loss

1.redis Master slave replication principle

1. Send a synchronization request from the library to the main library

2. The master library receives the synchronization request sent from the library

3. The main library is in use bgsave Generate rdb file

4. Main library rdb After the file is generated, it is saved to disk , Let go rdb The file is sent to the slave library

5. Receive from the main library rdb file , take rdb File loading memory

When the slave database synchronizes with the master database , It will empty all data from the library , So I'm doing redis When the master-slave, try to choose the one without any data redis

Architecture diagram

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_ Master slave copy

Environmental preparation

IP	service	role
192.168.81.210	redis-1	Main library
192.168.81.220	redis-2	Slave Library

2. Deploy two redis

2.1.192.168.81.210 To configure

1. establish redis Deployment path 
[[email protected] ~]# mkdir -p /data/redis_cluster/redis_6379/{conf,pid,logs,data}
[[email protected] ~]# tree /data/redis_cluster/
/data/redis_cluster/
└── redis_6379
    ├── conf
    ├── logs
    ├── pid
    └── data
    
2. download redis    
[[email protected] ~]# mkdir /data/soft
[[email protected] ~]# cd /data/soft
[[email protected] /data/soft]# wget https://repo.huaweicloud.com/redis/redis-3.2.9.tar.gz

3. Compilation and installation redis
[[email protected] /data/soft]# tar xf redis-3.2.9.tar.gz -C /data/redis_cluster/
[[email protected] /data/soft]# cd /data/redis_cluster/
[[email protected] /data/redis_cluster]# ln -s redis-3.2.9/ redis
[[email protected] /data/redis_cluster]# cd redis/src
[[email protected] /data/redis_cluster/redis]# make && make install

4. Prepare the configuration file 
[[email protected] ~]# vim /data/redis_cluster/redis_6379/conf/redis_6379.conf 
daemonize yes
bind 192.168.81.210 127.0.0.1
port 6379
pidfile /data/redis_cluster/redis_6379/pid/redis_6379.pid
logfile /data/redis_cluster/redis_6379/logs/redis_6379.log
databases 16
dbfilename redis_6379.rdb
dir /data/redis_cluster/redis_6379/data/
save 900 1
save 300 100
save 60 10000
appendonly yes
appendfilename "appendonly.aof"
appendfsync always
appendfsync everysec
appendfsync no
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes

5. start-up redis
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf 

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.
      22.
      23.
      24.
      25.
      26.
      27.
      28.
      29.
      30.
      31.
      32.
      33.
      34.
      35.
      36.
      37.
      38.
      39.
      40.
      41.
      42.
      43.
      44.
      45.
      46.
      47.

2.2.192.168.81.220 To configure

because redis-1 A set has been deployed redis, We can copy it directly

1. Use rsync take redis-1 Of redis Copy the directory and you 
[[email protected] ~]# rsync -avz /data [email protected]:/

2. View the copied directory file 
[[email protected] ~]# ls  /data/redis_cluster/
redis  redis-3.2.9  redis_6379
[[email protected] ~]# ls  /data/redis_cluster/redis_6379/
conf  data  logs  pid

3. Compilation and installation redis, Enable the system to use redis command 
 Direct execution make install that will do , Because the compilation step is redis-1 It has been done. 
[[email protected] ~]# cd /data/redis_cluster/redis-3.2.9/
[[email protected] /data/redis_cluster/redis-3.2.9]# make install

4. modify redis The configuration file 
[[email protected] ~]# vim /data/redis_cluster/redis_6379/conf/redis_6379.conf 
bind 192.168.81.220 127.0.0.1

5. start-up redis
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.

3.redis Master slave replication configuration

3.1. Batch create in the main library key

[[email protected] ~]# for i in {0..2000}; do redis-cli set k_${i} v_${i}; echo "k_${i} is ok"; done

127.0.0.1:6379> DBSIZE
(integer) 2001

     
      1.
      2.
      3.
      4.

3.2. Configure the slave library to connect to the master library

Two configuration methods are implemented Redis Master slave copy ：

Configuration file implementation
The command line executes the command to achieve

3.2.1. Set in the configuration file

1. Modify the configuration file 
[[email protected] ~]# vim /data/redis_cluster/redis_6379/conf/redis_6379.conf 
slaveof 192.168.81.210 6379				# Specifies the ip And port 

2. restart redis
[[email protected] ~]# redis-cli shutdown
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.

3.2.2. Hot update configuration on the command line

3.3. Check whether the slave database synchronizes the master database data

1. Execute... On the slave library keys Check whether the data is the same 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> keys * 

2. Create a new... On the main library key, Check whether the slave library exists 
 Lord ：
127.0.0.1:6379> set name jiangxl
OK
 from ：
127.0.0.1:6379> get name 
"jiangxl"

3. Delete... In the main library k_114, Check to see if it is also deleted from the library 
 Lord ：
127.0.0.1:6379> del k_114
(integer) 1
 from ：
127.0.0.1:6379> get k_114
(nil)


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_ data _02

3.4. The principle of master-slave synchronization in viewing logs

The master database and slave database have detailed logs recording replication

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_ data _03

4.redis Master slave copy dangerous operation

4.1. Misoperation using hot update configuration

redis If master-slave replication uses hot update configuration , Sometimes the slave database is mistaken for the master database because the wrong host is selected , As a result, on the main library slaveof, This will cause the data on the main database to be emptied , Because there is no data from the database

When the slave database synchronizes with the master database, it will empty all its own data

Misoperation process

 The data from the library is 0
127.0.0.1:6379> DBSIZE
(integer) 0
 The main database data is 2001
127.0.0.1:6379> DBSIZE
(integer) 2001

 Operate on the master database to synchronize the data of the slave database 
127.0.0.1:6379> SLAVEOF 192.168.81.220 6379
OK

 Look at the data again , The data has been cleared 
127.0.0.1:6379> DBSIZE
(integer) 0


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.

4.2. Avoid misoperation of hot update configuration

1. Do not use hot updates , Configure the master and slave directly in the configuration file

2. In execution slaveof When it's time to execute bgsave, Back up the data manually , Then in the data directory , take rdb Copy a file into another file , Make a backup , In this way, even if there is a problem, you can recover

bgsave After that, you don't have to restart , Direct backup rdb File can

1. Manual persistence 
127.0.0.1:6379> BGSAVE
Background saving started

2. Backup rdb file 
[[email protected] ~]# cd /data/redis_cluster/redis_6379/
[[email protected] /data/redis_cluster/redis_6379]# cd data/
[[email protected] /data/redis_cluster/redis_6379/data]# cp redis_6379.rdb redis_6379.rdb.bak

3. Synchronize the wrong main library again , Cause data loss 
127.0.0.1:6379> SLAVEOF 192.168.81.220 6379
OK
127.0.0.1:6379> keys *
(empty list or set)

4. Restore the backup rdb file , First stop redis, In restoring 
[[email protected] ~]# redis-cli shutdown
[[email protected] /data/redis_cluster/redis_6379/data]# cp redis_6379.rdb.bak redis_6379.rdb

5. View the restored data 
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf 
127.0.0.1:6379> DBSIZE
(integer) 2001


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.
      22.
      23.
      24.

5. simulation redis Master slave replication error data recovery

simulation redis Master slave synchronization misoperation data recovery

General train of thought ：

1. Empty two redis The data of

2. Create some data on the main library , And then use bgsave command , Save data to disk , Then put the disk rdb File backup

3. Then synchronize the data from the library , Simulation master database data loss

4. from rdb Backup file restore database

The main purpose of this experiment is to operate redis Backup restore

5.1. Empty data

Two sets of redis All need to be operated

First close, then delete data, and then start

[[email protected] ~]# redis-cli shutdown
[[email protected] ~]# rm -rf /data/redis_cluster/redis_6379/data/*
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf 

[[email protected] ~]# redis-cli
127.0.0.1:6379> keys *
(empty list or set)

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_ Master slave copy _04

5.2. Batch create data in the main library and back up

1. Batch create data 
[[email protected] ~]# for i in {0..2000}; do redis-cli set k_${i} v_${i}; echo "k_${i} is ok"; done
127.0.0.1:6379> DBSIZE
(integer) 2001

2. Write data to 
127.0.0.1:6379> BGSAVE
Background saving started

3. Backup rdb file 
[[email protected] /data/redis_cluster/redis_6379/data]# cp redis_6379.rdb redis_6379.rdb.bak
[[email protected] /data/redis_cluster/redis_6379/data]# ll
 Total usage  56
-rw-r--r--. 1 root root 27877 1 month   28 21:00 redis_6379.rdb
-rw-r--r--. 1 root root 27877 1 month   28 21:01 redis_6379.rdb.bak

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.

5.3. Synchronizing data from the library results in data loss

At this time, the data from the database should be empty

Master database synchronizes data from slave database , After synchronization, the master database data is lost , In this way, the data loss of the main database is simulated

5.4. Restore the data of the main library

Turn off first redis, Restore , Finally, open redis

1. Turn off the redis
[[email protected] ~]# redis-cli shutdown

2. Restore rdb Backup file 
[[email protected] /data/redis_cluster/redis_6379/data]# cp redis_6379.rdb.bak redis_6379.rdb
cp： Is it covered? "redis_6379.rdb"？ y

3. start-up redis
[[email protected] ~]# redis-server /data/redis_cluster/redis_6379/conf/redis_6379.conf 

4. See if the data is restored 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> keys *

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_ Master slave copy _05

6. Simulated online environment, main library downtime and fault recovery

Ideas ：

1. First, ensure that the master-slave synchronization has been configured , Both master and slave databases have data

2. close redis Main library , simulation redis Main library down

3. see redis Whether data still exists from the library , Whether it is readable or writable （ Can't write , Only read ）

4. Turn off the slaveof, Stop master-slave synchronization , Connect the application redis Change your address to from library , Ensure business continuity

5. Repair the main library , After the main library goes online , Establish a master-slave replication relationship with the slave Library , The original slave Library （redis-2） It becomes the main library , Now the master library becomes the slave Library （redis-1） At this time Close the application , Stop data writing

6. Then the main library will now （redis-2） Data synchronization to the current slave Library （redis-1）

7. Close now from the library （redis-1） Of slaveof, Stop master-slave replication , Then the main library will now （redis-2） To configure salveof, Synchronize the original master library （redis-1）

8. Data synchronization finished , The original master database is restored from the database

A simple and clear sentence ： The main library is down for some reason and cannot provide services , Directly switch from the library to the main library to provide services , Then, after the original primary database is restored, synchronize the data of the current primary database , Then stop all programs running online , Synchronize the current primary database with the restored primary database , Regenerate the master-slave relationship .

6.1. Configure the master-slave analog online environment

Before configuring master-slave, ensure that there is data on the master database

1. Log in to the main library redis-1 Check for data 
[[email protected] ~]# redis-cli
127.0.0.1:6379> DBSIZE
(integer) 2001

2. Log in from the library redis-2 Check for data 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> keys *
(empty list or set)

3. When there is no data in the slave library, configure the master-slave on the slave library , Synchronize the data of the main database 
127.0.0.1:6379> SLAVEOF 192.168.81.210 6379
OK
 The data has been synchronized 
127.0.0.1:6379> DBSIZE 
(integer) 2001

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.

6.2. Simulate the downtime of the master library and verify whether the slave library is readable and writable

1. Just close the main library , Cause downtime 
[[email protected] ~]# redis-cli shutdown

2. Check whether the slave library is readable and writable 
 Only read , Can't write 
[[email protected] ~]# redis-cli get k_1
"v_1"
[[email protected] ~]# redis-cli set k999 k_1
(error) READONLY You can't write against a read only slave.


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.

Once the main library goes down , The slave library will always output a log indicating that it is unable to connect to the main library

Redis Master slave replication cluster and recovery of abnormal data loss #yyds Dry inventory #_redis_06

6.3. Close the master-slave replication of the slave library to ensure uninterrupted business

Now the main library is not available , You can only read from the library, not write , But there's only one piece of data , We can only turn off master-slave replication on the slave Library , Make the slave library become the master library , Reconfigure the business redis Address , First of all, we must ensure that the business is not interrupted

1. Close slave Library redis-2 Master slave synchronization configuration of , Make it the main library 
[[email protected] ~]# redis-cli slaveof no one
OK

2. To be applied redis Change the address to from the library , Just turn off the master-slave configuration from the library , He himself is a readable and writable Library , The library has all the data before the failure , You can first ensure the uninterrupted business 

     
      1.
      2.
      3.
      4.
      5.

6.4. Repair the failed master database and synchronize the data of the original slave database

After repairing the main library , There are already data , Why synchronize data from the library , Because the moment the main library hangs up , The master-slave configuration is removed from the library , I have become the main library , It also provides data writing for a period of time , At this time, the data from the library is the most complete

Synchronize the main library now （ The original slave Library ） When the data is , First turn off the app , Don't write data in

In the main warehouse （ The original from the library ） Write several new data , New data generated by simulation

Configure master-slave synchronization on the main library that is going online again , Make yourself from the library , Synchronize the main library's （ The original slave Library ） data

After synchronization , Now from the library （ The repaired main database already has the latest data ）

 Stop the application before synchronizing , Don't go to redis Write data in 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> SLAVEOF 192.168.81.220 6379
OK
127.0.0.1:6379> get zuixinshujv
"vvvvv"


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.

6.5. Go back online from the main library

The "re online from the database" here refers to the original failed main database , Now it has been synchronized to the latest data , Therefore, it is necessary to go online and become the main database , He was chosen as the master library because his performance is higher than that of the slave library , Avoid future failures due to performance , So switch

1. Close slave Library （ The original main library ） The master-slave configuration of 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> SLAVEOF no one
OK

2. In the main warehouse （ The original slave Library ） Configure master-slave replication 
[[email protected] ~]# redis-cli 
127.0.0.1:6379> SLAVEOF 192.168.81.210 6379
OK

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.

6.6. To be applied redis The address is changed to the address of the main library again

At present, the main database has been restored , And the master-slave synchronization relationship is re established before the master-slave , Now you can apply redis The address is modified to be the main library , Just start the application

原网站

版权声明
本文为[Jiangxiaolong's technology blog]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/02/202202141055325984.html