Reading redis source code (V) master-slave replication and sentinel mechanism

Redis Single node has a single point of failure , To solve the single point problem , You need to be right Redis Configure slave , Then use the sentry to monitor the survival status of the master node , If the master node fails , The slave node can continue to provide caching function .

One 、 Master slave copy

Master-slave replication is generally used to realize data read-write separation , The master node provides write operations , Provides read operations from the node , It is suitable for reading more and writing less .

Field analytical ：

• runId： Every Redis Node startup will generate unique uuid, Every time Redis After restart ,runId Will change .
• offset： Both master and slave nodes maintain their own master-slave replication offsets offset When the master node has a write command ,offset=offset+ The byte length of the command ; The slave node receives the command sent by the master node , Will also increase their own offset, And put your own offset Send to master . such , The master node also saves its own offset And from the node offset, by force of contrast offset To determine whether the master-slave node data is consistent .
• repl_backlog_size： A fixed length first in, first out queue stored on the primary node , The default size is 1MB.

A. The copying process

（1） Execute... From node slaveof [masterIP] [masterPort], Actively connect to the primary server to request data synchronization .

slaveof The processing function of the command is replicaofCommand：

void replicaofCommand(client *c) {
    // slaveof no one Command can cancel the copy function 
    if (!strcasecmp(c->argv[1]->ptr,"no") &&
        !strcasecmp(c->argv[2]->ptr,"one")) {
    } else {
        //  Record the... Of the master server Ip Address and port 
        server.masterhost = sdsnew(ip);
        server.masterport = port;
        server.repl_state = REPL_STATE_CONNECT;
    }
    addReply(c,shared.ok);
}

（2） Discover the master node information from the scheduled task in the node , Build and master Socket Connect .

replicaofCommand The function simply records the primary server IP Address and port , No connection request to primary server .

The connection establishment is asynchronous , The related operations of master-slave replication are in the time event handler function serverCron In the ：

//  Perform master-slave replication related operations in one second cycle 
run_with_period(1000) replicationCron();

stay replicationCron in , The slave server sends a connection request to the master server ：

if (server.repl_state == REPL_STATE_CONNECT) {
    //  In connection , The corresponding file event will be created 
    if (connectWithMaster() == C_OK) {
        serverLog(LL_NOTICE,"MASTER <-> REPLICA sync started");
        server.repl_state = REPL_STATE_CONNECTING;
    }
}

（3） Send... From node Ping The signal , The master node returns Pong, Both sides can communicate with each other .

Ping package ：ping A bag consists of a bag head (type Field is CLUSTERMSG_TYPE_PING(0)) And multiple gossip section ( Recorded status information about other nodes , Include node name 、IP Address 、 Status and listening address , wait ) form .Redis Each node in the cluster can know the current state of other nodes through the heartbeat packet and save it to the state of this node .

Pong package ：pong Package format is the same as ping package , Just Baotou type The field is written as CLUSTERMSG_TYPE_PONG(1).

Be careful pong Packets except when received ping Bao He meet The packet will then be sent as a reply packet , After the master-slave switch , The new master node will send a message directly to all nodes in the cluster pong package , Notify the master-slave switch of the node role conversion .

（4） After the connection is established , Send from node to master node psync Command request to synchronize data , The master node sends all the data to the slave node （ Data synchronization ）.

The master server handles psync The entry function of the command is syncCommand.

【Redis 2.8 Then use psync [runId] [offset] command ; Support full and partial replication ;

Redis 4.0 Two more optimizations are proposed for master-slave replication , Put forward psync2 agreement 】

psync2 His advantage is that redis After the master-slave switch , There is no need to redo again fullsync Sync , Only partial synchronization is required , It's kind of similar binlog That kind of

（5） The master node synchronizes the current data to the slave node , Then the process of creating replication is completed ; then , The master node will continuously send the write command to the slave node , Ensure master-slave data consistency .

Each time the master server receives a write command request , Will broadcast the command request to all slave servers , At the same time, it is recorded in the copy buffer . The implementation function of broadcasting the command request to the slave server is replicationFeedSlaves, The logic is as follows ：

void replicationFeedSlaves(list *slaves, int dictid, robj **argv, int argc) {
   // If it is not equal to the last selected database , You need to synchronize first select command 
   if (server.slaveseldb != dictid) {
     // take select Command to add to the copy buffer 
     if (server.repl_backlog)
        feedReplicationBacklogWithObject(selectcmd);
     // Send... To all from servers select command 
     while((ln = listNext(&li))) {
       addReply(slave,selectcmd);
     }
   }
   server.slaveseldb = dictid;
   if (server.repl_backlog) {
     // Add the current command request to the copy buffer 
   }
​
   while((ln = listNext(&li))) {
     // Request synchronization commands from all slave servers 
  }
}

B. Copy in full

️ Send... From node psync ? -1 command （ Because for the first time , Do not know the master node's runId, So for ?, Because it's the first time to copy , therefore offset=-1）.

️ The master node finds that the slave node is the first replication , return FULLRESYNC {runId} {offset},runId It's the master node runId,offset Is the current offset.

️ After receiving the master node information from the node , Save to info in .

️ The master node is sending FULLRESYNC after , start-up bgsave command , Generate RDB file （ Data persistence ）.

️ The master node sends RDB File to slave . The write command of the master node is put into the buffer during the period of loading data from the slave node .

️ Clean up your own database data from the node .

️ Load from node RDB file , Save the data to your own database

️ The master node sends the write command in the buffer to the slave node

C. Incremental replication

️ Part of the replication is mainly Redis An optimization measure for the high cost of full replication , Use psync [runId] [offset] Command implementation .

When the slave node is replicating the master node , If there is an abnormal situation such as network flash or command loss , The slave node will request the master node to reissue the lost command data , The replication backlog buffer of the master node sends this data directly to the slave node .

In this way, you can maintain the consistency of the master-slave node replication . This part of data reissued is generally far less than the full amount of data .

️ The master node still responds to the command when the master-slave connection is interrupted , But the copy connection interrupt command cannot be sent to the slave node , However, the replication backlog buffer in the primary node can still store the latest write command data .

️ When the master-slave connection is restored , Since you saved yourself from the node Copied offset and The operation of the master node ID. So they will be treated as psync Parameters are sent to the master node , Request for Partial reproduction .

️ The master node received psync After the command, first check the parameters runId Is it consistent with itself , If the same , Note that the current primary node is copied before .

Then according to the parameters offset Find... In the replication backlog buffer , If offset The data after that exists , Then send... To the slave node +COUTINUE command , Indicates partial replication is possible . Because the buffer size is fixed , If there is a buffer overflow , Then make a full copy .

️ The master node sends the data in the replication backlog buffer to the slave node according to the offset , Make sure the master-slave copy is in normal state

int masterTryPartialResynchronization(client *c) {
    // Judge whether the server is running ID match , Whether the copy offset is legal 
    if (strcasecmp(master_replid, server.replid) &&
       (strcasecmp(master_replid, server.replid2) ||
        psync_offset > server.second_replid_offset))
    {
        goto need_full_resync;
    }
​
    // Determine whether the copy offset is included in the copy buffer 
    if (!server.repl_backlog ||
        psync_offset < server.repl_backlog_off ||
        psync_offset > (server.repl_backlog_off +   
               server.repl_backlog_histlen))
    {
        goto need_full_resync;
    }
    // Partial resynchronization , Identify the slave server 
    c->flags |= CLIENT_SLAVE;
    c->replstate = SLAVE_STATE_ONLINE;
    c->repl_ack_time = server.unixtime;
    // Add the client to the slave server linked list slaves
    listAddNodeTail(server.slaves,c);
​
   // According to the ability of the slave server +CONTINU
    if (c->slave_capa & SLAVE_CAPA_PSYNC2) {
        buflen = snprintf(buf,sizeof(buf),"+CONTINUE %s\r\n", server.replid);
    } else {
        buflen = snprintf(buf,sizeof(buf),"+CONTINUE\r\n");
    }
    if (write(c->fd,buf,buflen) != buflen) {
    }
    // Send the command request in the copy buffer to the client 
    psync_len = addReplyReplicationBacklog(c,psync_offset);
    // Update the number of valid slave servers 
    refreshGoodSlavesCount();
    return C_OK; /* The caller can return, no full resync needed. */
​
need_full_resync:
    return C_ERR;
}

Two 、 Sentinel mechanism

Once the primary node goes down , From node to master node , At the same time, you need to modify the address of the primary node of the application , You also need to command all slaves to copy the new master , The whole process needs human intervention . therefore , The sentinel mechanism can be used to manage this process .

The sentry is Redis High availability solution for , Can be in Redis Master Automatically select one in case of failure Redis Slave Switch to a Master, Continue to provide external services .

Redis Sentinel The minimum configuration is one master and one slave , The system can perform the following four tasks ：

• monitor ： Constantly check whether the master server and the slave server are running normally .
• notice ： When someone is being monitored Redis Server problem ,Sentinel adopt API Scripts notify administrators or other applications .
• Automatic failover ： When the master node is not working properly ,Sentinel An automatic failover operation is initiated , It will upgrade one of the slave nodes in the master-slave relationship with the failed master node to a new master node , And points the other slave nodes to the new master node .
• Configuration provider ： stay Redis Sentinel In mode , When the client application is initialized, it connects to Sentinel Node collection , Get information about the master node from .

A. Typical configuration file

//  Monitor a named mymaster Of Redis Master service , The address and port number are 127.0.0.1:6379,quorum by 2
sentinel monitor mymaster 127.0.0.1 6379 2 
​
//  If the sentry 60s Not received in mymaster Effective ping reply , Think mymaster be in down The state of 
sentinel down-after-milliseconds mymaster 60000
//  The timeout for switching is 180s
sentinel failover-timeout mymaster 180000
​
//  After switching , At the same time to the new Redis Master Initiating synchronous data request Redis Slave The number is 1
//  That is to say, after the switch is completed, let each Slave To synchronize data , Previous Slave Next after synchronization Slave To initiate a request to synchronize data 
sentinel parallel-syncs mymaster 1
​
//  Monitor a named resque Of Redis Master service , The address and port number are 127.0.0.1:6380,quorum by 4
// quorum:① take master Number of sentinels required to mark objective downline ;② The number of votes required by the election sentry to perform master-slave switchover 
sentinel monitor resque 192.168.1.3 6380 4

B. working principle

① Every Sentinel Nodes need to perform the following tasks on a regular basis ： Every Sentinel At a rate of once per second , To the master server it knows 、 From the server and other Sentinel Instance sends a PING command .

② If one slave Distance from the last valid reply PING The order took longer than down-after-milliseconds Specified value , So this slave Will be Sentinel Mark as subjective offline .

③ If one master Marked as subjective offline , So we're monitoring this master All of the Sentinel node , Confirm... At a rate of once per second master It has entered the subjective offline state .

④ If one master Marked as subjective offline , And there are enough Sentinel（ At least the number specified in the configuration file ） Agree to this judgment within a specified time frame , So this master Marked as objective offline .

⑤ In general , Every Sentinel With every 10 Second frequency to all that it knows mater and slave send out INFO command .

When one master When marked as objective offline ,Sentinel Go offline master It belongs to slave send out INFO The frequency of the command , from 10 One second changed to 1 Seconds at a time .

⑥ all Sentinel Negotiate objective offline master The state of , If in the SDOWN state , Then vote to automatically select a new master node , Point the remaining slave nodes to the new master node for data replication .

⑦ When there is not enough Sentinel agree! master When offline ,master The objective offline status of will be removed .

When master Reorientation Sentinel Of PING When the command returns a valid reply , The subjective offline state of the primary server will be removed .

C. Code flow

（1） start-up ：redis-server

redis-server /path/to/sentinel.conf --sentinel

After executing the command , The main code logic executed is as follows ：

main(){
    ...
    // Check whether the sentinel mode 
    server.sentinel_mode = checkForSentinelMode(argc,argv);
    ...
    if (server.sentinel_mode) {
        initSentinelConfig();        //  Set the listening port to 26379
        initSentinel();              //  Change the command that the Sentry can execute . The Sentry can only execute a limited number of server commands , Such as ping,sentinel,subscribe,publish,info wait . This function also initializes the sentry 
    }
    ...
    sentinelHandleConfiguration();        //  Parse configuration file , To initialize 
    ...
    sentinelIsRunning();                //  Randomly generate one 40 Byte sentinel ID, Start log printing 
    ...
}

stay main In the main stream of a function , Just initialize it . The real operation of establishing command connection and message connection is in the scheduled task serverCron in ：

serverCron(){
  // ...
  
    //  In sentinel mode , Used to establish a connection , And regularly send heartbeat packets and collect information 
    if (server.sentinel_mode) sentinelTimer();
    
    // ...
}

This function The main function as follows ：

1. Establish command connection and message connection . After the message connection is established, you will subscribe to Redis Service sentinel:hello channel .
2. On the command connection every 10s send out info Command to collect information ; Every time 1s Send on command connection ping Command to detect survivability ; Every time 2s Publish a message on the command connection , The format of the message is as follows ： sentinel_ip,sentinel_port,sentinel_runid,current_epoch,master_name,master_ip,master_port,master_config_epoch The above parameters respectively represent the IP、 The sentry's port 、 Sentinel's ID（ As mentioned above 40 Random string of bytes ）、 Current era （ Used for election and master-slave switching ）、Redis Master The name of 、Redis Master Of IP、Redis Master The port of 、Redis Master Configuration era （ Used for election and master-slave switching ）.
3. Check whether the service is in the subjective offline state .
4. Check whether the service is in an objective offline state and requires master-slave switching .

（2） If you judge one Redis Master In an objective offline state , You need to start the master-slave switch

Redis The selection rules are as follows ：

1. If it's time to Slave In the subjective offline state , Cannot be selected .
2. If it's time to Slave 5s There is no effective reply within ping Command or disconnect from the primary server for too long , Cannot be selected .
3. If slave-priority by 0, Cannot be selected （slave-priority Can be specified in the configuration file . Positive integer , The lower the value, the higher the priority , When the specified as 0 when , Cannot be selected as the primary server ）.
4. In the balance Slave Compare priorities in , The higher priority is selected ; If the priority is the same , Those with larger copy offset are selected ; Otherwise, select the top... In alphabetical order Slave.

This status requires that the selected Redis Slave Switch to a Redis Master, That is to say, the sentry is going to Slave Send the following command ：

MULTI              // Start a transaction 
SLAVEOF NO ONE     // Turn off the replication function of the slave server , Turn it into a master server 
CONFIG REWRITE     // take redis.conf File rewriting ( The original configuration will be overwritten according to the current running configuration )
CLIENT KILL TYPE normal //  Close the client connected to the service ( After shutdown, the client will reconnect , Recapture Redis Master The address of )
EXEC               // Perform transactions 

The Sentry will send the command to switch the master server to other slave servers in turn , as follows ：

MULTI              // Start a transaction 
SLAVEOF IP PORT    // Set up the server to request data from the new master server  
CONFIG REWRITE     // take redis.conf File rewriting ( The original configuration will be overwritten according to the current running configuration )
CLIENT KILL TYPE normal // Close the client connected to the service ( After shutdown, the client will reconnect , Recapture Redis Master The address of )
EXEC               // Perform transactions 

meanwhile , In the process , The sentry follows the corresponding state transition .