The underlying principles of RDB persistence and AOF persistence of redis

Redis RDB and AOF Of

because Redis yes In-memory database , It stores its database state in Memory Inside , So if you don't want to save the database state stored in memory to disk Inside , So once the server process exits , The state of the database in the server will also disappear . To solve this problem ,Redis Provides RDB Persistence function and AOF Persistence Two ways

RDB Persistence

RDB What the persistence function generates RDB file It's a compressed Binary Pieces of , This file allows you to restore the build RDB File time database state

however Redis How the server saves and loads RDB Of documents ?

SAVE Command and BGSAVE The implementation of the command

save

SAVE Orders will Blocking Redis Server process , until RDB file establish Until it's over , During server process blocking , The server Can't process any command requests

BGSAVE

1. BGSAVE The command will derive a Subprocesses , Then the child process is responsible establish RDB file , Server process （ The parent process ） Continue processing command requests ：
2. BGSAVE During the execution of the order , Sent by the client SAVE The command will be rejected by the server . Server forbidden SAVE Command and BGSAVE Commands are executed at the same time to avoid the parent process （ Server process ） And subprocesses execute two rdbSave call , Prevention of competitive conditions .
3. stay BGSAVE During the execution of the order , Sent by the client BGSAVE The command will be rejected by the server , Because two... Are executed at the same time BGSAVE Commands also create competitive conditions

The underlying code

establish RDB The actual work of the document is carried out by rdb.c/rdbSave Function completion ,SAVE Command and BGSAVE The command calls this function in different ways

Server file loading status

• RDB The loading of the file is Automatically execute when the server starts Of , therefore Redis It's not dedicated to loading RDB The order of the document , as long as Redis The server detected RDB File exists , It will Automatically loaded RDB file , The server is loading RDB During the documentation , It's going to be stuck , Until the load is complete

redis.conf To configure BGsave

save 900 1
save 300 10
save 60 10000

As long as any one of the following three conditions is met ,BGSAVE The command will be executed ：

1. The server 900 In seconds , The database is at least 1 Time modification .
2. The server 300 In seconds , The database is at least 10 Time modification . The server 60 In seconds ,
3. The database is at least 10000 Time modification

RDB File structure

REDIS

RDB The beginning of the document is REDIS part , The length of this part is 5 byte , preserved “REDIS” Five characters . Through these five characters , When a program loads a file , Quickly check whether the loaded file is RDB file

db_version

The length is 4 byte , Its value is an integer represented by a string , This integer is recorded RDB Version number of the document

databases

databases Part contains zero or any number of databases , And in each database Key value pair data

EOF

EOF The length of the constant is 1 byte , This constant indicates that RDB The end of the body of the document , When the reader encounters this value , It knows that all key value pairs of all databases have been loaded .

check_sum

It's a 8 Signed integer with byte length , A check sum is saved , This check sum is that the program passes the REDIS、db_version、databases、EOF The content of four parts is calculated .

see RDB file

command

od -c dump.rdb

[email protected]:/data# od -c dump.rdb

AOF Persistence

except RDB Beyond persistence ,Redis It also provides AOF（Append Only File） Persistence function . And RDB persistent Materialize by saving Key value pair To record different database States ,AOF Persistence It's through preservation Redis What the server does Write orders To record the state of the database

Persistence

RDB The way to persist the database state is to mess Key value pair Save to RDB file in , and AOF The method of persisting and saving the database state is to execute the server SET Save commands to AOF file in . When the server starts , You can load and perform AOF The command saved in the file To restore the database state before the server is shut down

Advantages and disadvantages

advantage ：

1、 Only one file dump.rdb, Easy persistence .
2、 Good disaster tolerance , A file can be saved to a safe disk .
3、 Maximize performance ,（fork Function out of ) Sub process to complete the write operation , Let the main process continue processing commands , So it is IO Maximize . Use a single child process for persistence , The main process will not do anything IO operation , To ensure the redis A high performance

When the subprocess is backed up , The parent process will not have any io operation （ No modification or deletion will be written ）, Ensure the integrity of data backup .

4. When the data set is large , Than AOF It's more efficient to start .

shortcoming ：

1. When something goes wrong , You may lose your last backup

2. The memory occupied by the child process is the same as that of the parent process （ Big data ）, Can cause CPU burden . If the server memory is large, it can be ignored

3. Can't do real-time backup . Data security is low .RDB It's persistence at intervals , If between persistence redis failure , There will be data loss . So this way is more suitable when the data requirements are not rigorous )

AOF Implementation of persistence （ load ）

AOF The implementation of persistence function can be divided into Order to append （append）、 File is written to 、 File synchronization （sync） Three steps .

Order to append

After the server executes a write command , It will append the executed write command to the server state in the protocol format aof_buf End of buffer

AOF File writing and synchronization

Redis The server process is a The event loop （loop）, In this cycle Document events Be responsible for receiving command requests from clients , And send a command reply to the client .

Time event Is responsible for the implementation of such as serverCron Function this requires a function that runs at a fixed time

Document events

When processing file events, write commands may be executed , Make some content be appended to aof_buf In the buffer , So before the server ends one event loop at a time , It will call flushAppendOnlyFile function , Consider whether it is necessary to aof_buf The contents of the buffer are written and saved to AOF In the document ,flushAppendOnlyFile The behavior of the function is configured by the server appendfsync The value of the option determines

appendfsync Three options （ Strategy ）

AOF Data restoration of files

Restore database

AOF rewrite

reason

because AOF Persistence records the state of the database by saving the executed write commands , So as the server runs ,AOF There will be more and more content in the document , The size of the files will get bigger and bigger , If not controlled , Too big AOF The document is likely to be right Redis The server 、 Even the entire host computer has an impact , also AOF The larger the size of the file , Use AOF The more time it takes for a file to restore data .

AOF Implementation of file rewriting

although Redis Will generate new AOF file Replace used AOF file The function of is named “AOF File rewriting ”, But actually ,AOF File rewriting does not need to be done on existing AOF File for any read 、 Analyze or write operations , This function is achieved by reading The current database of the server State to achieve .

The rewritten AOF Why files can be smaller ？ There are the following reasons ：

1. The data that has timed out in the process is no longer written to the file .
2. old AOF File contains invalid command , Such as del key1、set a 111、set a 222 etc. . Rewriting is directly generated using in-process data , So new AOF Write command to keep only the final data in the file .
3. Multiple write commands can be combined into one , Such as lpush list a、lpush list b、 lpush list c Can be converted to ：lpush list a b c.

AOF Backstage rewrite

The subprocess goes on AOF During rewrite , Server process （ The parent process ） You can continue processing command requests .

The process has a copy of the data of the server process , Use subprocesses instead of threads , You can avoid using locks , Ensure data security .

Execute... In a child process AOF During rewrite , The server process needs to perform the following three tasks ：

1. Execute the command from the client .
2. Append the executed write command to AOF buffer .
3. Append the executed write command to AOF Rewrite buffer .

Advantages and disadvantages

advantage ：

1、 Data security ,aof Persistence can be configured appendfsync attribute , Yes always, Every time Command operations are recorded as aof Once in the file .
2、 adopt append Mode write file , Even if the server goes down , Can pass redis-check-aof Tools for data consistency .
3、AOF The mechanism rewrite（ rewrite ） Pattern .AOF The document was not rewrite Before （ When the file is too large, the command Do merge rewrite ）, You can delete some of these commands （ For example, misoperation flushall）)

shortcoming ：

1、AOF File than RDB The file is big , And the recovery speed is slow .
2、 When the data set is large , Than rdb Low starting efficiency .

General drawing