Redis database persistence

One 、Redis summary

• Redis It is a non relational database of high-level key value pairs , It is also a cache database

• Redis and memcache Very similar , The difference is ,Redis Data can be persisted , And there are a lot of data types to support

• Redis The data type of has string 、 list 、 aggregate 、 Ordered set , It supports computing the sum of collections on the server side 、 Payment and supplement （ Bad ） Set , It also supports a variety of sorting functions , therefore Redis It can also be regarded as a data structure server

Two 、Redis The persistence of

• Redis All the data of is stored in memory , If persistence is not configured ,Redis Data will be lost after restart , So it needs to be turned on Redis Persistence of , Save the data in the memory on the hard disk , stay Redis After reboot , Data can be restored from hard disk to memory , And this process is divided into two modes ：

Semi persistent mode ： Irregularly save the data in memory to the hard disk through asynchronous synchronization , Also called RDB Pattern

Full persistence mode ： Write every data change in memory to a append only file（AOF file ） Inside , Also called AOF Pattern

（1） The persistence process

1. The client sends a write operation to the database server , The written data is saved in the memory of the client
2. The database server receives the client write request , The data of the write operation is saved in the memory of the server
3. Server calls write This system process , Write the data in the memory to the hard disk , The data will go to the buffer in the system memory first
4. The system transfers the data in the buffer to the disk controller , The data will be stored in the disk cache
5. The disk controller writes data to the physical media of the disk , At this time, the data is really saved to the disk

The above five steps are the ideal data saving process , But in most cases , The actual server will have various failures , The following are two cases of successful persistence after failure

1. Redis Database failure , Just complete the first three steps above , Then you can complete persistence , The rest of the operation is done by the system

2. If the system fails , You must complete all the above five steps , Data can be persisted

These two situations only refer to the possible failures during the storage process , But in fact, the saved data may also be damaged , A certain recovery mechanism is needed

Redis There are two policy mechanisms to implement the five steps of persistence , Namely RDB and AOF, By configuring redis.conf Master configuration file to configure two persistence methods , When the two persistence methods are enabled at the same time ,AOF A little higher priority

（2）RDB Mechanism

• RDB The data saving mechanism of is actually to save the data in the form of snapshots on the disk , The meaning of this snapshot is similar to that of the virtual machine , It should be noted that each generation RDB The document will put the old RDB File topping , That is to say, there is only one RDB file

• RDB Persistence refers to writing a snapshot of a data set in memory to disk within a specified time interval , It's also Redis Default persistence method , This way is to write the data in memory to the binary file in the form of snapshot , The default file name is dump,rdb

• since RDB The saving mechanism of is to generate a snapshot of the data within a specified time interval to save , Then there must be a trigger mechanism , about RDB Come on ,Redis There are three mechanisms , Namely ：save、bgsave、 automation

-save Trigger mode

When there is a client to Redis After sending the order , The command will block Redis The server , In execution Save Mechanism （ Save data to RDB file ） In the process of ,Redis You cannot execute other commands , You can't save it until it's finished

When Redis In preservation RDB When you file , If the previously saved RDB file , that Redis The new document will replace the old one , In the real world ,Redis The received client requests may be tens of thousands or even hundreds of thousands , This approach is obviously not desirable

-bgsave Trigger mode

Use bgsave after , The operation sent by the client ,Redis Asynchronous snapshot saving will be performed in the background , and Redis You can also continue to accept the operations of other clients

When the client sends the operation to Redis Server time , stay Redis After the operation , Will execute fork（） Action create subprocess ,RDB The persistence operation of is completed by the child process , After the subprocess completes the operation , Will automatically end . The blocking in this process only occurs in the stage of the child process , Basically Redis Internally owned RDB The operation is to use bgsave Accomplished

- automation

Automation means automatic triggering , By Redis Primary profile for Redis.conf To complete , The following options can be configured ：

1. save：

This is used to configure trigger Redis The persistence condition of , That is, when to save the data in memory to the hard disk , The format is save m n, When m Within seconds n Changes in data , Just save the data to the hard disk , And the mechanism adopted is bgsave, If you don't need persistence, you can put all save Line comment out to disable this function

2. stop-writes-on-bgsave-error：

The default configuration of this option is yes, It means when it starts RDB And when the last background save of data failed ,Redis Whether to stop receiving data , To configure yes It can make managers realize that the data is not correctly persisted to the hard disk , Otherwise, configure no The management personnel will not be aware of the accident . After stopping receiving data, you only need to restart Redis, You can start receiving data again

3. rdbcompression

The default value for this option is yes, It means whether to compress and store snapshots stored on disk

4. rdbchecksum

The default value for this option is yes, It means after storing the snapshot , Whether to let redis Use crc64 Algorithm to check data , After being turned on, the performance consumption will increase by about 10% , if necessary redis Maximum performance , You can turn this option off

5. dbfilename

Configure the file name of the snapshot , The default is dump.rdb

6. dir

Set the path to save the snapshot , This path must be a directory , Not the file name , The default is to save to the current directory ./

contrast save and bgsave：

command	save	bgsave
Synchronization mode （IO type ）	Sync	asynchronous
Is it blocked	Blocking	Blocking （ Occurs in child processes ）
advantage	No extra memory consumed	Do not block clients , You can continue to receive commands
shortcoming	Block client command	Child processes are required , Memory consumption

-RDB Advantages and disadvantages of

• advantage ：

1. RDB The files are compact , And full backup , Ideal for backup and disaster recovery
2. In the use of bgsave Mechanism generation RDB When you file ,redis The main process will create a new sub process to handle all data saving work , The main process does not need any disk IO operation , And you can continue to receive operations from other clients
3. RDB It's faster to recover large data sets than AOF fast

• Inferiority ：

1. Generated RDB Snapshot is a full backup , What is stored is the binary serialization form of memory data , Very compact in storage .
2. During snapshot persistence , A child process will be started for snapshot persistence , During this period, the data modified by the main process will not be reflected in the child process , That is, the data during data persistence will not be saved , Data may be lost . And when the data set is large , It can cause the whole server to stop serving for hundreds of microseconds or even a second

（3）AOF Mechanism

• Full backup is always time consuming , and AOF Provides a more efficient way , Working mechanism is very simple ,redis I will pass every write command I receive write Function is appended to the specified file , This and mysql The binary log of is very similar

• AOF Persistence mode ：

whenever Redis When you receive a write command ,Redis It will record AOF In file

- File rewriting

AOF The persistence method of also brings a problem , persistent AOF The file will get bigger and bigger , To compress AOF Persistent files ,Redis Provides bgrewriteaof command , When saving the data in the current memory to a temporary file in the form of a command , At the same time, a new subprocess will be created to rewrite the file to a new AOF In file , This method is a bit similar to snapshot

-AOF Three trigger mechanisms

1. Every change is synchronized （always）： Synchronous persistence , Every time the data changes, it will be immediately recorded on the disk , Poor performance but good data integrity

2. A second synchronous （everysec）： Asynchronous synchronization , Every second , But if in a second Redis Server down , Then one second of data will be lost

3. No synchronization （no）： No synchronization , This means that the system decides when to synchronize

Compare the three mechanisms ：

command	always	everysec	no
advantage	No data loss	One synchronization per second	No management
shortcoming	Poor performance	At most one second of data will be lost	Uncontrollable

-AOF Advantages and disadvantages of

• advantage ：

1. AOF Better protect data from loss , commonly AOF Every second , Then perform a synchronization operation through a background thread , So at most one second of data is lost
2. AOF The log file does not have any disk addressing overhead （ The location of the log file is self specified ）, Very high write performance , The file is not easy to be damaged
3. AOF When there is a background rewrite operation in the log file , It will not affect the reading and writing of the client
4. AOF Log files record commands , This feature is very suitable for disaster recovery accidental deletion and emergency recovery , for example ： Someone accidentally used flushall Command clears all data , At this time, as long as the rewrite operation does not happen , You can copy it immediately AOF file , Delete the last item flushall Command and then AOF File back , Through the recovery mechanism , Recover database

• shortcoming ：

1. For the same data ,AOF Documents are usually better than RDB The snapshot file of is larger
2. AOF After opening , Supported, written QPS（ Query rate per second ） than RDB Written QPS low , because AOF Generally, it is configured to synchronize files every second , Of course , Synchronization every second , Performance consumption will not be very high

（4） choice RDB still AOF

1. Willing to sacrifice some performance , You can choose AOF
2. In exchange for higher cache consistency , Ensure the highest possible integrity of the database , choice AOF
3. Willing to write frequently , Do not enable backup for better performance , In manual operation save Make a backup when , choice RDB

Generally speaking, the effect of using the two together is the best , But if you open both ways at the same time , Only AOF Will take effect ,Redis By default RDB,AOF coordination RDB when , Both can be operated manually save

3、 ... and 、RDB Persistent configuration

Get ready Reids：

[[email protected] ~]# vim /usr/local/redis/redis.conf 
......
 217 
 218 save 900 1      # Turn on RDB after , stay 900 Within seconds 1 Changes in data trigger the generation of snapshots 
 219 save 300 10
 220 save 60 10000
 221 
......
 234 # permissions, and so forth.
 235 stop-writes-on-bgsave-error yes  # When RDM After the snapshot fails , It does not affect the user's write operation 
 236 
......
 240 # the dataset will likely be bigger if you have compressible values or keys.
 241 rdbcompression yes  # Whether or not to RDB Snapshot file compression , Performance will increase after shutdown 
 242 
......
 250 rdbchecksum no   # close RDB Check and verify snapshots , Will increase performance 
 251  
 252 # The filename where to dump the DB
 253 dbfilename dump.rdb  # The name of the snapshot 
 254 
......
 262 # Note that you must specify a directory here, not a file name.
 263 dir /usr/local/redis/rdb   # Storage path of snapshots 
 264 
......
# Save and exit 
[[email protected] ~]# mkdir /usr/local/redis/rdb
[[email protected] ~]# /etc/init.d/redis start # Turn on redis
/var/run/redis_6379.pid exists, process is already running or crashed
Redis is running...
[[email protected] ~]# redis -h 192.168.100.202 -p 6379 # Get into redis Writing data 
192.168.100.202:6379> auth 123123
OK
192.168.100.202:6379> set aaa bbb
OK
192.168.100.202:6379> set bbb ccc
OK
192.168.100.202:6379> keys *
1) "aaa"
2) "bbb"
192.168.100.202:6379> exit
[[email protected] ~]# /etc/init.d/redis restart # restart redis
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped
Starting Redis server...
1073:C 05 Jun 2021 21:54:27.156 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1073:C 05 Jun 2021 21:54:27.156 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1073, just started
1073:C 05 Jun 2021 21:54:27.156 # Configuration loaded
Redis is running...
[[email protected] ~]# redis -h 192.168.100.202 -p 6379 # Enter again to check whether the data is still there 
192.168.100.202:6379> auth 123123
OK
192.168.100.202:6379> keys * 
1) "aaa"
2) "bbb"
 # Data found to exist ,redis The data of is stored in memory , If persistence is not configured , restart redis The data is lost , But it turns on RDB Persistence mechanism , And every... Is configured in the configuration file 600 Take a snapshot when the data changes every second , Just now I wrote two data , After you've written Redis In fact, the snapshot has been taken and saved in the specified path , So restart Redis Post data still exists 
 192.168.100.202:6379> exit
[[email protected] ~]# ll /usr/local/redis/rdb/
 Total usage  4
-rw-r--r-- 1 root root 115 6 month    5 21:54 dump.rdb

Four 、AOF Persistent configuration

[[email protected]s7 ~]# vim /usr/local/redis/redis.conf 
......
 217 
 218 #save 900 1 # hold save To all comments means to close RDB Persistence mode 
 219 #save 300 10
 220 #save 60 10000
 221 
......
 262 # Note that you must specify a directory here, not a file name.
 263 dir /usr/local/redis/rdb   # This is also AOF File storage path 
 264 
......
 698 
 699 appendonly yes  # Turn on AOF Persistence mode 
 700 
 701 # The name of the append only file (default: "appendonly.aof")
 702 
 703 appendfilename "appendonly.aof"  # Set persistent files 
 704 
......
 728 # appendfsync always
 729 appendfsync everysec  # Set up AOF The way of persistence , The default is everysec Once a second 
 730 # appendfsync no
 731 
......
 750 
 751 no-appendfsync-on-rewrite no  # If set to yes, be redis The executed commands will be stored in the buffer , Wait for the system to automatically synchronize to the hard disk ,no Can 
 752 
......
 769 
 770 auto-aof-rewrite-percentage 100  # If currently written AOF The file has reached the size of the last rewritten file 100% Words , Trigger rewrite 
 771 auto-aof-rewrite-min-size 64mb  # Set up AOF The minimum size of the persistent rewrite file , When reach 64MB And in accordance with 100% when , The rewrite operation is triggered 
 772 
......
# Save and exit 
[[email protected] ~]# /etc/init.d/redis restart 
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Redis stopped
Starting Redis server...
1122:C 05 Jun 2021 22:08:19.456 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1122:C 05 Jun 2021 22:08:19.456 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1122, just started
1122:C 05 Jun 2021 22:08:19.456 # Configuration loaded
Redis is running...
[[email protected] ~]# redis -h 192.168.100.202 -p 6379
192.168.100.202:6379> auth 123123
OK
192.168.100.202:6379> keys *
(empty list or set)
192.168.100.202:6379> set aaa bbb
OK
192.168.100.202:6379> keys *
1) "aaa"
192.168.100.202:6379> set bbb ccc
OK
192.168.100.202:6379> keys *
1) "bbb"
2) "aaa"
192.168.100.202:6379> exit
[[email protected] ~]# cat /usr/local/redis/rdb/appendonly.aof # You can see AOF All the commands in the file are executed 
*2
$6
SELECT
$1
0
*3
$3
set
$3
aaa
$3
bbb
*3
$3
set
$3
bbb
$3
ccc