Redis high availability - master-slave replication, sentinel mode, cluster

---

One 、Redis High availability

1、 What is high availability

stay web Server , High availability refers to the time when the server can be accessed normally , The measure is how long a normal service can be provided (99.9%、99.99%、99.999% wait ).

But in Redis In context , The meaning of high availability seems to be broader , In addition to ensuring normal service （ Such as the separation of master and slave 、 Fast disaster recovery technology ）, Also need to consider the expansion of data capacity 、 Data security will not be lost .

2、Redis High availability technology

stay Redis in , Technologies for high availability include persistence 、 Master slave copy 、 Sentinels and cluster colony , Their functions are described below , And what kind of problems have been solved .

• Persistence ： Persistence is the simplest high availability method （ Sometimes it's not even classified as a highly available means ）, The main function is data backup , Store data on hard disk , Ensure that data is not lost due to process exit .

• Master slave copy ： Master slave replication is highly available Redis The basis of , Sentinels and clusters are highly available based on master-slave replication . Master-slave replication mainly realizes multi machine backup of data （ And synchronization ）, And load balancing and simple fault recovery for read operations .

  • defects ： Failure recovery cannot be automated ; Write operations are not load balanced ; Storage capacity is limited by a single machine .

• sentry ： On the basis of master-slave replication , Sentinels achieve automated recovery from malfunctions .（ The master is dead , Find a new master , Sentinel node to monitor ）

  • defects ： Write operations are not load balanced ; Storage capacity is limited by a single machine .

• Cluster colony ： By clustering ,Redis It solves the problem that write operation cannot be load balanced , And storage capacity is limited by single machine , A relatively complete high availability scheme has been realized .（6 Taiwan start , in pairs ,3 Lord 3 from ）

Persistence technology has been introduced in the previous article （ Redis High availability persistence - Nuggets (juejin.cn) ）, This article will be specific to master-slave replication 、 sentry 、Cluster Cluster three high availability technologies .

---

Two 、Redis Master slave copy

• Master slave copy , It means to put one Redis Server data , Copy to other Redis The server . The former is called the main node （Master）, The latter is called the slave node （slave）; Data replication is one-way , From master to slave only .
• By default , Each station Redis Servers are all primary nodes ; And a master node can have multiple slave nodes （ Or no slave ）, But a slave node can only have one master node .

1、Redis The role of master-slave replication

• data redundancy ： Master-slave replication realizes hot backup of data , It's a way of data redundancy beyond persistence .
• Fault recovery ： When there is a problem with the master node , Can be served by a slave node , Fast fault recovery ; It's actually a redundancy of services .
• Load balancing ： On the basis of master-slave replication , Cooperate with the separation of reading and writing , Write service can be provided by the master node , Read service provided by slave node ( The write Redis Connect master node when data is applied , read Redis Apply connection from node when data ), Share server load ; Especially in the situation of less writing and more reading , Sharing read load through multiple slave nodes , Can be greatly improved Redis Concurrency of servers .
• High availability cornerstone ： In addition to the above functions , Master slave replication is also the foundation for sentinels and clusters to implement , So master-slave replication is Redis High availability Foundation .

2、 Master slave replication process

（1） If you start a slave Machine processes , Then it will go to Master The machine sends a message sync command command , Request synchronous connection .

（2） Whether it's the first connection or reconnection ,Master The machine will start a background process , Save data snapshot to data file （ perform rdb operation ）, meanwhile Master All commands to modify data are also recorded and cached in the data file .

（3） After the background process completes the cache operation ,Master The machine will turn to slave The machine sends data files ,slave The end machine saves the data file to the hard disk , Then load it into memory , next Master The machine will send all operations to modify the data to slave End machine . if slave Failure causes downtime , It will automatically reconnect after returning to normal .

（4）Master The machine received slave After the connection of the end machine , Send its complete data file to slave End machine , If Mater Receive multiple at the same time slave Synchronization request from , be Master A process will be started in the background to save the data file , Then send it to all slave End machine , Ensure that all slave The end machine is normal .

---

3、 ... and 、 The construction of master-slave replication

Experiment preparation

master：20.0.0.12
slave1：20.0.0.6
slave2：20.0.0.5

1、 All host installations Redis

Refer to the last blog Redis Installation and deployment

2、 modify master The configuration file of the node (20.0.0.12)

vim /etc/redis/6379.conf
bind 0.0.0.0	
appendonly yes	

/etc/init.d/redis_6379 restart	


3、 Modify two slave node Redis The configuration file

vim /etc/redis/6379.conf
bind 0.0.0.0
replicaof 20.0.0.12 6379
appendonly yes

/etc/init.d/redis_6379 restart

4、 Verify the master-slave effect

## stay master View the log on the node 
tail -f /var/log/redis_6379.log 

## stay master Create data 
redis-cli
set name run
get name

## View on the slave node 
redis-cli
get name

---

Four 、Redis Sentinel mode

The core function of the sentry : On the basis of master-slave replication , Sentinel introduces automatic failover of the primary node .

1、 The principle of sentinel mode

It's a distributed system , It is used to monitor each server in the master-slave structure , When In case of failure, choose a new one through the voting mechanism master, And put all slave Connect to the new master. So the whole cluster of sentinels The quantity shall not be less than 3 Nodes .

2、 The role of sentinel mode

• monitor ： The Sentry will constantly check whether the master and slave nodes are working properly .
• Automatic failover ： When the master node is not working properly , The Sentry will start an automatic failover operation , It will upgrade one of the failed master nodes from a node to a new master , And make the other slave nodes copy the new master node .
• notice （ remind ）： Sentinels can send failover results to clients .

3、 The structure of sentinel mode

• The sentinel node ： The sentinel system consists of one or more sentinel nodes , Sentinel nodes are special redis node , Don't store data .

• Data nodes ： Both master and slave nodes are data nodes .

The start of the sentry depends on the master-slave mode , Therefore, we must install the master-slave mode before doing sentinel mode . Sentinel mode needs to be deployed on all nodes , Sentinel mode will monitor all Redis Whether the work node is normal , When Master When something goes wrong , Because other nodes lose contact with the primary node , So I will vote , Half the vote thinks this Master There is a problem , Then the sentry room will be notified , And then from Slaves Select one of them as the new Master.

Here's the thing to watch out for , Objective offline is the only concept of the master node ; If the slave and sentinel nodes fail , By sentinels subjective offline , There will be no further objective offline and failover operations .

---

5、 ... and 、 build Redis Sentinel mode

Then the master-slave replication experiment above is done

1、 Modify the configuration files of the three sentinel nodes

vim /opt/redis-5.0.7/sentinel.conf
protected-mode no								
port 26379										
daemonize yes									logfile "/var/log/sentinel.log"					
dir "/var/lib/redis/6379"						
sentinel monitor mymaster 20.0.0.12 6379 2			

2、 Start sentinel mode

## Start three sentinels 
## First start master, To rev. slave
cd /opt/redis-5.0.7/
redis-sentinel sentinel.conf &

3、 fault simulation

## see redis-server Process number 
ps -ef | grep redis
## Kill master Node redis-server The process number of 
kill -9   Process number 	   

redis-cli -p 26379 info Sentinel 

---

6、 ... and 、Redis Cluster mode

• colony , namely Redis Cluster, yes Redis 3.0 The introduction of distributed storage scheme .

• A cluster consists of multiple nodes (Node) form ,Redis Data is distributed among these nodes .

• The nodes in the cluster are divided into master and slave nodes ： Only the master node is responsible for the maintenance of read-write requests and cluster information ; The slave node only copies the master node data and status information .

1、 The role of clusters

（1） Data partition ： Data partition ( Or data fragmentation ) It is the core function of cluster .
Clusters spread data across multiple nodes , On the one hand, it has broken through Redis The limit of single machine memory size , There's a huge increase in storage capacity ; On the other hand, each master node can provide external reading and writing services , It greatly improves the response ability of the cluster .
Redis The size of single machine memory is limited , Both persistence and master-slave replication are mentioned ; for example , If the single machine memory is too large ,bgsave and bgrewriteaof Of fork Operation may cause the main process to block , When the host is switched in the master-slave environment, the slave node may be unable to provide services for a long time , During the full replication phase, the replication buffer of the master node may overflow .

（2） High availability ： Cluster supports master-slave replication and automatic fail over of master nodes （ Like a sentry ）; When any node fails , Clusters can still provide external services

By clustering ,Redis It solves the problem that write operation cannot be load balanced , And storage capacity is limited by single machine , A relatively complete high availability scheme has been realized .

2、Redis Data fragmentation of cluster

Redis Cluster introduces the concept of hash slot .

Redis Cluster has 16384 Hash slot （ Number 0-16383）.
Each node of the cluster is responsible for a portion of the hash slot .

Every Key adopt CRC16 Check pair 16384 Take the rest to decide which Hashi slot to place , By this value , Go to find the node corresponding to the corresponding slot , Then directly and automatically jump to the corresponding node for access operation .

 With 3 Take a cluster of nodes as an example ：
 node A contain 0 To 5460 Hash slot 
 node B contain 5461 To 10922 Hash slot 
 node C contain 10923 To 16383 Hash slot 

3、 The master-slave replication model of cluster mode

• There are A、B、C Three nodes , If node B failed , The whole cluster will be short of 5461-10922 This range of slots can not be used .
• Add a slave node for each node A1、B1、C1 The whole cluster has three Master Nodes and three slave Node composition , At the node B After failure , A cluster election B1 Bit continues to serve as the master node . When B and B1 After failure , The cluster will not be available .

---

7、 ... and 、 build Redis Cluster mode

Experiment preparation

master1：20.0.0.12
slave1：20.0.0.5
master2：20.0.0.7
slave2：20.0.0.6
master3：20.0.0.10
slave3：20.0.0.11

6 All the tables are installed redis

1、 Turn on the cluster function

cd /opt/redis-5.0.7/
vim redis.conf
......
bind 20.0.0.12                    #69 That's ok , Modify it to monitor your physical network card IP
protected-mode no                         #88 That's ok , It is amended as follows no, Turn off protection mode 
port 6379                                 #92 That's ok ,redis Default listening port 
daemonize yes                             #136 That's ok , Start Daemons , Start as a separate process 
appendonly yes                            #699 That's ok , It is amended as follows yes, Turn on AOF Persistence 
cluster-enabled yes                       #832 That's ok , uncomment , Turn on the cluster function 
cluster-config-file nodes-6379.conf       #840 That's ok , uncomment , Cluster name file settings 
cluster-node-timeout 15000                #846 That's ok , uncomment , Cluster timeout settings 

## Pass the document to another 5 Nodes , Then each node should modify the listening address to its own IP
scp redis.conf 20.0.0.5:`pwd`
scp redis.conf 20.0.0.6:`pwd`
scp redis.conf 20.0.0.7:`pwd`
scp redis.conf 20.0.0.10:`pwd`
scp redis.conf 20.0.0.11:`pwd`

And so on , Total to 5 Servers ;
Then each node should modify the listening address to its own IP Address

cd /opt/redis-5.0.7/
vim redis.conf

2、 All nodes start redis service

redis-server redis.conf

3、 Start cluster

## Start the cluster at any node 
redis-cli --cluster create 20.0.0.12:6379 20.0.0.7:6379 20.0.0.10:6379 20.0.0.5:6379 20.0.0.6:6379 20.0.0.11:6379 --cluster-replicas 1
## The first three are master, The last three are slave

4、 Test cluster

# Add -c Parameters , Nodes can jump to each other   
redis-cli -h 20.0.0.12 -p 6379 -c 
# View the hash slot number range of the node 
cluster slots    
# assignment 
set name run
# Check the hash slot number of the key   
cluster keyslot  Key name 
​
​
redis-cli -h 20.0.0.12 -p 6379 -c
20.0.0.12:6379> cluster slots      # View the hash slot number range of the node 
1) 1) (integer) 10923         # The hash slot number range of the first pair of master-slave 
   2) (integer) 16383
   3) 1) "20.0.0.10"     # Master node 
      2) (integer) 6379
      3) "5f128a6e204d1d6f6dc924ad8b39034a8e9f3261"
   4) 1) "20.0.0.11"     # From the node 
      2) (integer) 6379
      3) "4a05a086eec06fa4da58b15512d1c81184bc5ee5"
2) 1) (integer) 5461          # The hash slot number range of the second pair of master-slave 
   2) (integer) 10922
   3) 1) "20.0.0.7"     # Master node 
      2) (integer) 6379
      3) "3008bba29dfbf342bc448ba3062b0a331c8d009e"
   4) 1) "20.0.0.6"     # From the node 
      2) (integer) 6379
      3) "ee61a4709d6420bb540b2c28218fdd2dfe358b7a"
3) 1) (integer) 0             # The hash slot number range of the third pair of master-slave 
   2) (integer) 5460
   3) 1) "20.0.0.12"     # Master node 
      2) (integer) 6379
      3) "d1ddb554b3edaebefa6672b2f1f8171393e1f7f3"
   4) 1) "20.0.0.5"     # From the node  
      2) (integer) 6379
      3) "71e1f705ce01ca31ab16fa3cf07d7e6cbfab5978"
20.0.0.12:6379>
​
​
# stay 12 New node name key , Will automatically jump to 7 Node for storage      
20.0.0.12:6379> set name run
-> Redirected to slot [5798] located at 20.0.0.7:6379
OK
20.0.0.7:6379> cluster keyslot name    # see name Hash slot number of key 
(integer) 5798
20.0.0.7:6379> quit       # Exit database 
[[email protected] ~]# redis-cli -h 20.0.0.12 -p 6379 -c     # Log back in 10 node 
20.0.0.12:6379> keys *     #12 There is no name key 
(empty list or set)
20.0.0.12:6379> get name    # see name Value of key , It will automatically jump to... According to the hash slot number of the key 7 Node to get 
-> Redirected to slot [5798] located at 20.0.0.7:6379
"run"
20.0.0.7:6379>            # Has jumped to 7 node