Implement redis sentinel to simulate master failure scenarios

1. Experimental framework

2. Configure one master and two slave environments

1. One click compile install redis6.2.4
 The specific methods ： See https://blog.csdn.net/aa896517050/article/details/125955201

2. Execute at all master and slave nodes ,
[[email protected]:~]#
sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e 's/^# masterauth.*/masterauth 123456/' -e 's/^# requirepass .*/requirepass 123456/' /apps/redis/etc/redis.conf

3. Execute at all slave nodes , The master node can be configured without replicaof 10.0.0.7 6379
[[email protected]:~]#
echo "replicaof 10.0.0.8 6379" >> /etc/redis.conf

4. All slave nodes start redis
[[email protected]:~]#
systemctl enable --now redis

3. Achieve sentinel （ Each host operates the same ）

1. Source code compilation requires copying sentinel.conf File to /apps/redis/etc/ Next ,yum Installed directly on /etc/ under .
[[email protected]:~]#
cd redis-6.2.4
[[email protected]:~/redis-6.2.4]#
cp sentinel.conf  /apps/redis/etc/redis-sentinel.conf

2. Modify the configuration file sentinel.conf
[[email protected]:~/redis-6.2.4]#
vim /apps/redis/etc/redis-sentinel.conf
# protected-mode no
bind 0.0.0.0  # Where to add 
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379

daemonize no   # The normal version is modified to yes, Background operation , however 6.2.4 It won't start , It will automatically turn off , It must be changed to no Talent .


pidfile "/apps/redis/run/redis-sentinel.pid"  # Appoint pid File path 

logfile "/apps/redis/log/sentinel_26379.log"   # Specify the log file path 

dir /tmp

sentinel monitor mymaster 10.0.0.7  6379 2 # modify ip
...
sentinel auth_pass mymaster 123456   # Add verification password 
...
sentinel down-after-milliseconds mymaster 3000
...
acllog-max-len 128
...
sentinel parallel-syncs mymaster 1
...
sentinel failover-timeout mymaster 180000
...
sentinel deny-scripts-reconfig yes
...
SENTINEL resolve-hostnames no
...
SENTINEL announce-hostnames no
...

3. Examination result 
[[email protected]:~/redis-6.2.4]#
grep -Ev "^#|^$" /apps/redis/etc/redis-sentinel.conf 
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel_26379.log"
dir /tmp
sentinel monitor mymaster 10.0.0.7  6379 2
sentinel auth-pass mymaster 123456
sentinel down-after-milliseconds mymaster 3000
acllog-max-len 128
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
SENTINEL resolve-hostnames no
SENTINEL announce-hostnames no

3. Be sure to modify permissions , otherwise redis-sentinel.conf No authority , Can't start . Because of moving root jurisdiction , No redis jurisdiction .sentinel_26379.log I can't check the log ,
[[email protected]:~/redis-6.2.4]#
chown -R redis.redis /apps/redis/  （ Every node has to execute ）


4. Make sentinel Of service file （ Every mainframe ）
[[email protected]:~/redis-6.2.4]#
vim /lib/systemd/system/redis-sentinel.service

[Unit]
Description=Redis Sentinel
After=network.target

[Service]
ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd
ExecStop=/bin/kill -s QUIT $MAINPID
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

5. Restart the host （ Every mainframe ）
[[email protected]:~/redis-6.2.4]#
systemctl enable --now  redis-sentinel

6. Check startup status 
[[email protected]:~/redis-6.2.4]#
ss -ntl 
State   Recv-Q Send-Q Local Address:Port  Peer Address:Port  
LISTEN      0      511     *:26379               *:*                  
LISTEN      0      511     *:6379                *:*                  
LISTEN      0      128     *:22                  *:*                  
LISTEN      0      100    127.0.0.1:25           *:*                  
LISTEN      0      511     [::1]:6379            [::]:*                  
LISTEN      0      128      [::]:22              [::]:*                  
LISTEN      0      100     [::1]:25              [::]:*                  

4. Simulate the fault and detect the result

1. The master node simulates a fault 
127.0.0.1:6379> shutdown

2. Check the log 
# View the log of the primary node that has been down 
[[email protected]:~]#
tail -f  /apps/redis/log/sentinel_26379.log
20554:X 23 Jul 2022 23:50:14.203 # +sdown master mymaster 10.0.0.7 6379
20554:X 23 Jul 2022 23:50:14.353 # +new-epoch 1
20554:X 23 Jul 2022 23:50:14.356 # +vote-for-leader 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 1
20554:X 23 Jul 2022 23:50:15.310 # +odown master mymaster 10.0.0.7 6379 #quorum 3/2
20554:X 23 Jul 2022 23:50:15.310 # Next failover delay: I will not start a failover before Sat Jul 23 23:56:15 2022
20554:X 23 Jul 2022 23:50:15.475 # +config-update-from sentinel 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 10.0.0.27 26379 @ mymaster 10.0.0.7 6379
20554:X 23 Jul 2022 23:50:15.475 # +switch-master mymaster 10.0.0.7 6379 10.0.0.17 6379
20554:X 23 Jul 2022 23:50:15.475 * +slave slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.17 6379
20554:X 23 Jul 2022 23:50:15.475 * +slave slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379
20554:X 23 Jul 2022 23:50:18.493 # +sdown slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379

# Promote the slave node log of the master node 
[[email protected]:~]#
tail -f  /apps/redis/log/sentinel_26379.log
10346:X 23 Jul 2022 23:50:14.317 # +sdown master mymaster 10.0.0.7 6379
10346:X 23 Jul 2022 23:50:14.353 # +new-epoch 1
10346:X 23 Jul 2022 23:50:14.357 # +vote-for-leader 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 1
10346:X 23 Jul 2022 23:50:14.371 # +odown master mymaster 10.0.0.7 6379 #quorum 3/2
10346:X 23 Jul 2022 23:50:14.372 # Next failover delay: I will not start a failover before Sat Jul 23 23:56:15 2022
10346:X 23 Jul 2022 23:50:15.477 # +config-update-from sentinel 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 10.0.0.27 26379 @ mymaster 10.0.0.7 6379
10346:X 23 Jul 2022 23:50:15.477 # +switch-master mymaster 10.0.0.7 6379 10.0.0.17 6379
10346:X 23 Jul 2022 23:50:15.477 * +slave slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.17 6379
10346:X 23 Jul 2022 23:50:15.477 * +slave slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379
10346:X 23 Jul 2022 23:50:18.503 # +sdown slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379

# It is still the log of the master node 
[[email protected]:~]#
tail -f  /apps/redis/log/sentinel_26379.log
9913:X 23 Jul 2022 23:50:14.289 # +sdown master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.342 # +odown master mymaster 10.0.0.7 6379 #quorum 2/2
9913:X 23 Jul 2022 23:50:14.342 # +new-epoch 1
9913:X 23 Jul 2022 23:50:14.342 # +try-failover master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.345 # +vote-for-leader 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 1
9913:X 23 Jul 2022 23:50:14.356 # a970a9757ba57b6dfd793936b8397a3bf5d6f6cb voted for 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 1
9913:X 23 Jul 2022 23:50:14.357 # e549971186fcf36e3f2b3510298e4503f1a21b9b voted for 8dfdc855fc3949309a1e4e1d880de1dfad5c5dd3 1
9913:X 23 Jul 2022 23:50:14.429 # +elected-leader master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.430 # +failover-state-select-slave master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.482 # +selected-slave slave 10.0.0.17:6379 10.0.0.17 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.482 * +failover-state-send-slaveof-noone slave 10.0.0.17:6379 10.0.0.17 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:14.544 * +failover-state-wait-promotion slave 10.0.0.17:6379 10.0.0.17 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:15.415 # +promoted-slave slave 10.0.0.17:6379 10.0.0.17 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:15.415 # +failover-state-reconf-slaves master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:15.473 * +slave-reconf-sent slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:16.443 * +slave-reconf-inprog slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:16.444 * +slave-reconf-done slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:16.506 # -odown master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:16.506 # +failover-end master mymaster 10.0.0.7 6379
9913:X 23 Jul 2022 23:50:16.506 # +switch-master mymaster 10.0.0.7 6379 10.0.0.17 6379
9913:X 23 Jul 2022 23:50:16.506 * +slave slave 10.0.0.27:6379 10.0.0.27 6379 @ mymaster 10.0.0.17 6379
9913:X 23 Jul 2022 23:50:16.506 * +slave slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379
9913:X 23 Jul 2022 23:50:19.508 # +sdown slave 10.0.0.7:6379 10.0.0.7 6379 @ mymaster 10.0.0.17 6379

3. The original master node is restarted , By default, join the cluster as a slave node .
[[email protected]:~/redis-6.2.4]#
systemctl restart redis
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.0.0.17
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:431717
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:fa3a0489ac02dfcb7ad477e60ad35ba026d64d11
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:431717
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:428218
repl_backlog_histlen:3500