Redis source code analysis RDB

We knew that in the third grade of primary school ,redis Is a pure memory storage middleware , So what about it ？ Will the data be lost ？ The answer is not to lose . in fact redis In order to ensure that the data will not be lost in case of downtime , Two mechanisms for data persistence are provided ——rdb and aof.

rdb On a regular basis will be the full amount of data in memory dump To disk , The next time you start, you can load the previous data directly ,rdb The problem is that it can only provide a snapshot of the data at a certain time , There is no guarantee that the data will not be lost after the snapshot is established , therefore redis It also provides aof.aof All the way through Append Only File, Its principle is to write all the changes to disk one by one . In this blog, let's focus on rdb Implementation of persistence ,aof Keep it for the next blog .

rdb Related to the source code

stay redis in , Trigger rdb There are several ways to save it .

save command

We are redis-cli Next, call save The command will trigger rdb File generation , If the child process is not generating in the background rdb, Will call rdbSave() Generate rdb file , And save it on disk .

void saveCommand(client *c) {
    //  Check whether there is a process executing in the background save, If there is, stop executing .
    if (server.child_type == CHILD_TYPE_RDB) {
        addReplyError(c,"Background save already in progress");
        return;
    }
    rdbSaveInfo rsi, *rsiptr;
    rsiptr = rdbPopulateSaveInfo(&rsi);
    if (rdbSave(server.rdb_filename,rsiptr) == C_OK) {
        addReply(c,shared.ok);
    } else {
        addReplyErrorObject(c,shared.err);
    }
}

Redis Of rdbSave Function is really going on RDB Persistent functions , Its general process is as follows ：

1. First, create a temporary file .
2. Create and initialize rio,rio yes redis Yes io An abstraction of , Provides read、write、flush、checksum…… Other methods .
3. call rdbSaveRio(), Will the current Redis Write the full amount of memory information to the temporary file .
4. call fflush、 fsync and fclose Interface writes files to disk .
5. Use rename Change the name of the temporary file to ceremonial RDB file .
6. take server.dirty Zero clearing ,server.dirty It's a record that was generated last time rdb How many data changes since then , Will be in serverCron It is used in .

The specific code is as follows ：

/* rdb Disk write operation  */
int rdbSave(char *filename, rdbSaveInfo *rsi) {
    char tmpfile[256];
    char cwd[MAXPATHLEN]; /* Current working dir path for error messages. */
    FILE *fp = NULL;
    rio rdb;
    int error = 0;

    snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
    fp = fopen(tmpfile,"w");
    if (!fp) {
        char *cwdp = getcwd(cwd,MAXPATHLEN);
        serverLog(LL_WARNING,
            "Failed opening the RDB file %s (in server root dir %s) "
            "for saving: %s",
            filename,
            cwdp ? cwdp : "unknown",
            strerror(errno));
        return C_ERR;
    }

    rioInitWithFile(&rdb,fp);  //  initialization rio,
    startSaving(RDBFLAGS_NONE);

    if (server.rdb_save_incremental_fsync)
        rioSetAutoSync(&rdb,REDIS_AUTOSYNC_BYTES);
    //  Memory data dump To rdb 
    if (rdbSaveRio(&rdb,&error,RDBFLAGS_NONE,rsi) == C_ERR) {
        errno = error;
        goto werr;
    }

    /*  Swipe the data to disk and delete it , Make sure there is no data left in the operating system buffer  */
    if (fflush(fp)) goto werr;
    if (fsync(fileno(fp))) goto werr;
    if (fclose(fp)) { fp = NULL; goto werr; }
    fp = NULL;
    
    /*  Rename the temporary file to the official file name  */
    if (rename(tmpfile,filename) == -1) {
        char *cwdp = getcwd(cwd,MAXPATHLEN);
        serverLog(LL_WARNING,
            "Error moving temp DB file %s on the final "
            "destination %s (in server root dir %s): %s",
            tmpfile,
            filename,
            cwdp ? cwdp : "unknown",
            strerror(errno));
        unlink(tmpfile);
        stopSaving(0);
        return C_ERR;
    }

    serverLog(LL_NOTICE,"DB saved on disk");
    server.dirty = 0;
    server.lastsave = time(NULL);
    server.lastbgsave_status = C_OK;
    stopSaving(1);
    return C_OK;

werr:
    serverLog(LL_WARNING,"Write error saving DB on disk: %s", strerror(errno));
    if (fp) fclose(fp);
    unlink(tmpfile);
    stopSaving(0);
    return C_ERR;
}

because redis It is a single-threaded model , So in save Can't handle requests in the process of , The single threaded model can save There will be no data changes in the process of data mining , but save It could last a long time , This can lead to redis Unable to process read and write requests properly , It's very lethal for online services , therefore redis It also provides bgsave command , It can be executed without affecting normal reading and writing save operation , Let's look at the concrete implementation .

bgsave command

bgsave Provides background generation rdb File functionality ,bg Meaning is background, How to achieve it ? In fact, it's called fork() Generated a subprocess , And then it's done in the subprocess save The process of .

void bgsaveCommand(client *c) {
    int schedule = 0;

    /* The SCHEDULE option changes the behavior of BGSAVE when an AOF rewrite
     * is in progress. Instead of returning an error a BGSAVE gets scheduled. */
    if (c->argc > 1) {
        if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"schedule")) {
            schedule = 1;
        } else {
            addReplyErrorObject(c,shared.syntaxerr);
            return;
        }
    }

    rdbSaveInfo rsi, *rsiptr;
    rsiptr = rdbPopulateSaveInfo(&rsi);

    if (server.child_type == CHILD_TYPE_RDB) {
        addReplyError(c,"Background save already in progress");
    } else if (hasActiveChildProcess()) {
        if (schedule) {
            server.rdb_bgsave_scheduled = 1;  //  If bgsave It's already in execution , This execution will be put on serverCron In the implementation of 
            addReplyStatus(c,"Background saving scheduled");
        } else {
            addReplyError(c,
            "Another child process is active (AOF?): can't BGSAVE right now. "
            "Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenever "
            "possible.");
        }
    } else if (rdbSaveBackground(server.rdb_filename,rsiptr) == C_OK) {
        addReplyStatus(c,"Background saving started");
    } else {
        addReplyErrorObject(c,shared.err);
    }
}

int rdbSaveBackground(char *filename, rdbSaveInfo *rsi) {
    pid_t childpid;

    if (hasActiveChildProcess()) return C_ERR;

    server.dirty_before_bgsave = server.dirty;
    server.lastbgsave_try = time(NULL);
    //  Create child process ,redisFork Actually, it's about fork Encapsulation  
    if ((childpid = redisFork(CHILD_TYPE_RDB)) == 0) {
        int retval;

        /*  Subprocesses  */
        redisSetProcTitle("redis-rdb-bgsave");
        redisSetCpuAffinity(server.bgsave_cpulist);
        retval = rdbSave(filename,rsi);
        if (retval == C_OK) {
            sendChildCowInfo(CHILD_INFO_TYPE_RDB_COW_SIZE, "RDB");
        }
        exitFromChild((retval == C_OK) ? 0 : 1);
    } else {
        /*  The parent process  */
        if (childpid == -1) {
            server.lastbgsave_status = C_ERR;
            serverLog(LL_WARNING,"Can't save in background: fork: %s",
                strerror(errno));
            return C_ERR;
        }
        serverLog(LL_NOTICE,"Background saving started by pid %ld",(long) childpid);
        server.rdb_save_time_start = time(NULL);
        server.rdb_child_type = RDB_CHILD_TYPE_DISK;
        return C_OK;
    }
    return C_OK; /* unreached */
}

bgsave It's just a way of save The process is executed in the subprocess , This way, the parent process won't be blocked . There was a problem when I first saw this , How does the subprocess save a snapshot at a certain time when the parent process is continuously reading and writing memory ？ This redid There's no special treatment in , It depends on the operating system fork(). When a process calls fork() when , The operating system copies a copy of the current process , Including the contents of memory in the current process . So it can be said that as long as fork() success , The data in the current memory is fully copied . Of course, the specific implementation of the kernel in order to enhance fork() Performance of , Used copy-on-write Technology , Only when the copied data is changed by the parent process or child process will it be copied .

serverCron

Generated above rdb Both of them are passively triggered ,redis It also provides regular generation rdb The mechanism of .redis About rdb The generated configuration is as follows ：

save <seconds> <changes> 
##  for example 
save 3600 1 # 3600 If there is 1 If you write a book, you will create rdb
save 300 100 # 300 If there is 100 If you write a book, you will create rdb
save 60 10000 # 60 If there is 1000 If you write a book, you will create rdb

Generated periodically rdb Implementation in server.c Medium serverCron in .serverCron yes redis Once at a time eventloop Scheduled tasks that are executed on a regular basis , There is rdb and aof Execution logic ,rdb The details are as follows ：

int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
    /*
    .  Leave out the rest of the code 
    */
    /*  testing bgsave and aof Whether the rewrite is in progress  */
    if (hasActiveChildProcess() || ldbPendingChildren())
    {
        run_with_period(1000) receiveChildInfo();
        checkChildrenDone();
    } else {
        /* If there is not a background saving/rewrite in progress check if
         * we have to save/rewrite now. */
        for (j = 0; j < server.saveparamslen; j++) {
            struct saveparam *sp = server.saveparams+j;

            /*  Check to see if execution is achieved save Standards for  */
            if (server.dirty >= sp->changes &&
                server.unixtime-server.lastsave > sp->seconds &&
                (server.unixtime-server.lastbgsave_try >
                 CONFIG_BGSAVE_RETRY_DELAY ||
                 server.lastbgsave_status == C_OK))
            {
                serverLog(LL_NOTICE,"%d changes in %d seconds. Saving...",
                    sp->changes, (int)sp->seconds);
                rdbSaveInfo rsi, *rsiptr;
                rsiptr = rdbPopulateSaveInfo(&rsi);
                rdbSaveBackground(server.rdb_filename,rsiptr);
                break;
            }
        }
    }
    /*
    .  Leave out the rest of the code 
    */
    /*  If last triggered bgsave There is already a process in progress , It will mark rdb_bgsave_scheduled=1, Then put serverCron
     *  In the implementation of  
     */
    if (!hasActiveChildProcess() &&
        server.rdb_bgsave_scheduled &&
        (server.unixtime-server.lastbgsave_try > CONFIG_BGSAVE_RETRY_DELAY ||
         server.lastbgsave_status == C_OK))
    {
        rdbSaveInfo rsi, *rsiptr;
        rsiptr = rdbPopulateSaveInfo(&rsi);
        if (rdbSaveBackground(server.rdb_filename,rsiptr) == C_OK)
            server.rdb_bgsave_scheduled = 0;
    }
    /*
    .  Leave out the rest of the code 
    */
}

rdb File format

rdb The specific file format is relatively simple , As follows ：

----------------------------#
52 45 44 49 53              #  Magic  "REDIS"
30 30 30 33                 # ASCII code rdb Version number of  "0003" = 3
----------------------------
FA                          #  Auxiliary field 
$string-encoded-key         #  May contain multiple meta information  
$string-encoded-value       #  such as redis Version number , Creation time , Memory usage ……...
----------------------------
FE 00                       # redis db Number . db number = 00
FB                          #  identification db Size 
$length-encoded-int         # hash The size of the table (int)
$length-encoded-int         # expire hash The size of the table (int)
----------------------------#  From here, that's the beginning k-v data  
FD $unsigned-int            #  How many seconds does the data have to expire (4byte unsigned int)
$value-type                 #  identification value data type (1 byte)
$string-encoded-key         # key,redis String type (sds)
$encoded-value              # value,  The type depends on  $value-type
----------------------------
FC $unsigned long           #  How many milliseconds does the data have to expire (8byte unsigned long)
$value-type                 #  identification value data type (1 byte)
$string-encoded-key         # key,redis String type (sds) 
$encoded-value              # value,  The type depends on  $value-type
----------------------------
$value-type                 # redis data key-value, No expiration time 
$string-encoded-key
$encoded-value
----------------------------
FE $length-encoding         # FE Identify the previous db End of data , And then add the length of the data  
----------------------------
...                         #  other redis db Medium k-v data , ...
FF                          # FF rdb The end of the file   
8-byte-checksum             ##  And finally 8byte Of CRC64 The checksum  

summary

rdb To a certain extent, it guarantees redis The instance does not lose data in case of abnormal downtime , When it's generated on a regular basis rdb snapshot , Changes after snapshot generation cannot be appended to rdb In file , therefore rdb There is no complete guarantee that data will not be lost , So redis It also provides another data persistence mechanism aof, We'll see in the next article . in addition , In execution bgsave It's highly dependent on the operating system fork() Mechanism , This also brings a lot of performance overhead , See Linux fork Hidden overhead - Out of date fork( main story line subject under discussion )

Reference material

1. Redis RDB Persistent details
2. https://rdb.fnordig.de/file_format.html
3. Redis Persistence
4. Linux fork Hidden overhead - Out of date fork( main story line subject under discussion )

This article is about Redis Source analysis series blog , At the same time, there are also corresponding Redis Chinese annotation version , I want to learn more about it Redis Classmate , welcome star And attention . Redis Chinese annotation version warehouse ：https://github.com/xindoo/Redis Redis Source analysis column ：https://zxs.io/s/1h If you found this article useful , welcome One key, three links .