Redis master-slave realizes 10 second check and recovery

Redis Master-slave framework

problem ：redis down After restart , Can recover key value , Time will be delayed ,down Values between machines will be lost .

Experimental environment , The master and slave are on one The server On , Use different ports .

Solution ：redis The master-slave is still not perfect , We make use of redis The master and slave mainly do redis Backup ,master After downtime , Realization 10s Recover as much as possible within key value , The main job is reading and writing , Achieve fast read and write , Do not do any backup , Simple and practical from rdb Way to achieve backup .

The principle of thinking ：

redis There are two ways to persist ,rdb And aof Pattern , in consideration of aof The mode grows too fast , Recovery is slow , Just do it from the top rdb Pattern , Lord down after , The script checks the state of the master , Then do it from the top bgsave, Put the top rdb file copy To the master data Directory , Restart the main . By judging the Lord key Worth the quantity , Whether it is the same as the quantity on , To determine whether it returns to normal , And then you can do it key Worth a full recovery . The backup method is , An hour copy Once from the top rbd file , Make an hourly backup .

The idea of script It mainly realizes the above ideas and principles , The script is a little awkward ,2 Be the master ,1 It's done redis from . Awkward , experiment , There is no time to revise

#!/bin/bash

DATE=`date +%Y%m%d%M%s`

REDIS_DIR=/usr/local/redis

REDIS2_DIR=/usr/local/redis2

BIN_DIR=/usr/local/redis/bin

DATA1_DIR=/usr/local/redis/data

DATA2_DIR=/usr/local/redis2/data

PORT1=6378

PORT2=6379

function BACKUP_RDB {

    ps -ef |grep $PORT2 >/dev/null

    if  [ $? -eq 0 ];then

        ${BIN_DIR}/redis-cli -p $PORT2 bgsave >/dev/null

        if [ $? -eq 0 ];then

            echo -e "\033[31m MASTER REDIS DOWN AND SLAVE bgsave DONE\033[0m"

            ${BIN_DIR}/redis-cli -p $PORT2 SLAVEOF NO ONE >/dev/null

        fi

    fi

}

function CP_RDB {

    [ -d ${DATA1_DIR}/redisbackup ]  || mkdir ${DATA1_DIR}/redisbackup

    if [ -f $DATA2_DIR/dump.rdb ];then

        mv ${DATA1_DIR}/dump.rdb  ${DATA1_DIR}/redisbackup/dump.rdb.$DATE

        cp ${DATA2_DIR}/dump.rdb  ${DATA1_DIR}/

    fi

}

function  START_REDIS {

    KEY2_NUM=`${BIN_DIR}/redis-cli -p $PORT2 INFO  |grep db[0-9]*.key |awk -F[:=,] '{total+=$3}END{print total}'`

    ${BIN_DIR}/redis-server ${REDIS_DIR}/etc/redis.conf

    netstat -tnlp |grep "$PORT1" >/dev/null

    a=$?

    if [ $a -ne 0 ];then

        while [ $a -ne 0 ];do

            sleep 2

            netstat -tnlp | grep "127.0.0.1:$PORT1" 1>/dev/null 2>/dev/null

            a=$?

        done

        echo -e "\033[32m MASTER REDIS RUNNING ........\033[0m"

        sleep 7

        KEY1_NUM=`${BIN_DIR}/redis-cli -p $PORT1  INFO  |grep db[0-9]*.key |awk -F[:=,] '{total+=$3}END{print

total}'`

#        echo ${KEY1_NUM}

        if [ $KEY1_NUM -ge $KEY2_NUM ];then

            ${BIN_DIR}/redis-cli -p $PORT2 SLAVEOF 127.0.0.1 $PORT1 >/dev/null

            echo -e "\033[32m MASTER AND SLAVE are Normal sync \033[0m"

        else

            echo -e "\033[31m KEY NUM are not equal, MASTER AND SLAVE are not Normal sync,place to check ! \033[0m"

            break

        fi

    else

        echo -e "\033[32m MASTER REDIS RUNNING ........\033[0m"

        sleep 7

        KEY1_NUM=`${BIN_DIR}/redis-cli -p $PORT1 INFO  |grep db[0-9]*.key |awk -F[:=,] '{total+=$3}END{print total}'`

#        echo ${KEY1_NUM}

        if [ $KEY1_NUM -ge $KEY2_NUM ];then

            ${BIN_DIR}/redis-cli -p $PORT2 SLAVEOF 127.0.0.1 $PORT1 >/dev/null

            echo -e "\033[32m MASTER AND SLAVE are Normal sync \033[0m"

        else

            echo -e "\033[31m KEY NUM are not equal, MASTER AND SLAVE are not Normal sync,place to check ! \033[0m"

            break

        fi

    fi

}

while true;do

    sleep 10

    netstat -tnlp | grep "127.0.0.1:$PORT1" 1>/dev/null 2>/dev/null

    TRAP1=$?

    ps -ef |grep "$PORT1" >/dev/null

    TRAP2=$?

    if [ $TRAP1 -ne  0 ] || [ $TRAP2 -ne  0 ];then

        #echo redis-master down

        BACKUP_RDB

        CP_RDB

        START_REDIS

    fi

done

In the process of recovery , Compare the main key And the slave after the Master goes down key, To determine whether it returns to normal ,

Problems encountered ： In the process of recovery , Lord key The value of is always less than that of key Value , But the main startup is completely normal ,key The value is exactly the same , After thinking, I found , It is the problem of program execution , Script execution is too fast , When it comes to comparison ,key The number of values has not fully recovered , It's just a matter of time , So it's implemented sleep 3 Seconds later , Everything is all right . Everyone should according to their own key It is worth the size of the quantity to determine the judgment sleep Time for .

If there's a better one redis Master-slave implementation , Please share .