Persistence practice of redis (Linux version)

Redis Persistence practice (Linux edition )

1. Introduction to persistence

Using permanent storage media to save data , The working mechanism of restoring the saved data at a specific time is called persistence .
Prevent accidental loss of data , Ensure data security

2.RDB Related configuration

The following and save What's relevant is 1234;
and bgsave What's relevant is 12345;
and RDB The manual start mode is related to 12346;( although RDB The manual starting mode is RDB, But with 5 irrelevant )
and AOF What's relevant is 2789;
and AOF Manual rewriting is related to 2789;
and AOF Manual automatic rewriting is related to 2789 10 11;

1.dbfilename dump.rdb：
explain ： Set the local database file name , The default value is dump.rdb
Experience ： Usually set to dump- Port number .rdb
2.dir：
explain ：rdb Path to file
Experience ： It is usually set to a directory with large storage space , Directory name data
3.rdbcompression yes：
explain ： Set whether to compress data when storing to local database , The default is yes, use LZF Compress
Experience ： It is usually on by default , If set to no, Can save CPU The elapsed time , But it makes the stored files bigger （ huge ）
4.rdbchecksum yes：
explain ： Set whether to RDB File format verification , The verification process is carried out in the process of writing and reading files
Experience ： It is usually on by default , If set to no, It can save the reading and writing process about 10% Time consuming , However, there is a certain risk of data corruption
5.stop-writes-on-bgsave-error yes：
explain ： If there is an error in the background stored procedure , Whether to stop the save operation
Experience ： Usually, it defaults to open
6.save second changes：
effect : Meet the time limit key When the number of changes reaches the specified number, persistence is carried out
Parameters :
second： Monitor the time frame
changes： monitor key Amount of change
7.appendonly no：
appendonly yes|no
Open or not AOF Persistence function , The default is not on
8.appendfsync everysec：
appendfsync always|everysec|no
AOF Write data strategy , Default everysec
9.appendfilename “appendonly.aof” ：
appendfilename filename
AOF Persistent filename , The default file name is not appendonly.aof, Recommended configuration is appendonly- Port number .aof,AOF The path to save the persistent file is the same as RDB Persistent files are consistent
10.auto-aof-rewrite-percentage 100：
auto-aof-r ewrite-percentage percentage
11.auto-aof-rewrite-min-size 64mb：
auto-aof-rewrite-min-size size
Refers to the percentage of automatic rewriting

download redis Post default configuration ：

3.RDB Manual start mode -save

download redis Post default data directory ：

Delete both

Preservation , There will be another dump.rdb file

save

Save again , Although it is a binary file , But it can be vaguely found rdb The documents have changed

Turn off the server

Open the server again

The client can still view the key we just saved

4.RDB Manual start mode -save working principle

The order is as follows , return “save”

save

Be careful ：save The execution of the instruction blocks the current Redis The server , Until now RDB Until the process is complete , It may cause long-term congestion , Online environment is not recommended ( If the following save If it takes too long ,get Will block )

5.RDB Manual start mode -bgsave working principle

The order is as follows , return “Background saving started”

bgsave

For resolution save The directive “ Too much data , How to deal with the low efficiency caused by single thread execution ” problem ;
Start the background save operation manually , But not immediately ;
bgsave The order is for save Optimization of blocking problem .Redis It's all about RDB operation （ namely “RDB Automatic start mode - Modify the configuration ”） All use bgsave The way ,save The command can be discarded ,bgsave And save I'm using one rdb file

6.RDB Manual start mode -bgsave

Then bgsave operation , Discovery tips “Background saving started”

see dump.rdb, Vaguely found that there was a beijing

7.RDB Automatic start mode - Modify the configuration

Related to the following configurations ：
save second changes

8.RDB Automatic start mode - Modify how the configuration works

1." Auto start mode configuration " It should be set according to the actual business situation , If the frequency is too high or too low, performance problems will occur , The results can be catastrophic ：
For example, don't set it to “ Two key Change once and store once ” Frequency too high
2." Auto start mode configuration " For in the second And changes Settings usually have complementary correspondence , Try not to set up an inclusive relationship ：
Such as the first “ The setting time is short and the amount of change is small , Set to 1 Second change 1 Time key”, Set another Cheng “100 Second change 100 Time key”,“100 Second change 100 Time key” Will be “1 Second change 1 Time key” To limit , So generally second When I was a child changes Just set it big ,second When I was big changes Just set small
3." Auto start mode configuration " After startup, execute bgsave operation ：
save 900 1 The meaning is ： When the time comes 900 seconds , If redis The data happened, at least 1 Changes , execute bgsave

Each instruction will return a result to redis, The following three conditions must be met ：
1. Will have an impact on the data ：
such as get The operation has no effect
2. It really made an impact ：
If there is an impact on the data, but the data value does not change , It's not called “ It really made an impact ”, The value must be changed before it really has an impact
3. No data comparison ：
That is, if we treat the same key Twice in a row set Instructions , There will be no data comparison ,redis Think it's two key There is a change

9.RDB Comparison of three startup modes

notes :”RDB Automatic start mode ： And “bgsave” equally , Omit here

10.RDB Special starting form

10.1 Copy in full

Copy in full :
Explain in detail in master-slave replication

10.2 Restart the server while the server is running

If you restart the server while the server is running , It will start automatically RDB preservation

debug reload

10.3 Specify to save data when shutting down the server while the server is running

If you restart the server while the server is running , It will start automatically RDB preservation
Execute by default shutdown On command , Automatic execution bgsave( If it's not on AOF Persistence function ）

shutdown save

11.RDB Advantages and disadvantages

RDB advantage :
1.RDB Is a compact binary , High storage efficiency
2.RDB What's stored inside is redis A snapshot of data at a point in time , Perfect for data backup , Full scale replication and other scenarios
3.RDB It's faster to recover data than AOF Much faster
4. application ： Every X Hours to perform bgsave Backup , And will RDB Copy the file to the remote machine , For disaster recovery .

RDB shortcoming
1.RDB Whether it's executing instructions or using configuration , Unable to achieve real-time persistence , There is a high probability of data loss
2.bgsave The instructions are executed every time they run fork Action create subprocess , To sacrifice some performance , Extra consumption of memory
3.Redis In many versions of RDB Unified version of file format , There may be incompatibility of data formats among different versions of services
such as ：2.0redis Of rdb file 4.0 I can't read it , In fact, there are solutions , stay 2.0 After saving the data to the file through the program , Then restore it to 4.0 The kind of data source format ,4.0 You can't read it
4. A large amount of data is stored , Low efficiency , Based on the idea of snapshot , Every read and write is all data , When the amount of data is huge , Very inefficient
5. Big data IO Low performance
6. The risk of data loss caused by downtime

12.AOF Introduce

about RDB The shortcomings of ,AOF Make corrections ,AOF The solution is as follows ：
1. Don't write full data , Only part of the data is recorded
2. Reduce the difficulty of distinguishing whether the data has changed , Change record data to record operation process
3. Record all operations , Eliminate the risk of data loss

AOF(append only file) Persistence ： Each write is logged as a separate log , Reexecute on reboot AOF The command in the file achieves the purpose of recovering data . And RDB In contrast, it can simply describe the process of changing recorded data to recorded data generation
AOF Its main function is to solve the real-time problem of data persistence , So far Redis The mainstream way of persistence

13.AOF How to write data

When the server receives the instructions we executed , The server did not immediately record aof In the document , Instead, it was placed in a temporary area , namely “AOF Write command to be operated ” The corresponding buffer , After a certain stage , Synchronize the commands in the past buffer to aof Just in the file

14.AOF Three strategies for writing data (appendfsync)

1.always( Every time ）
Each write operation is synchronized to AOF In file , Data zero error , Low performance , Not recommended
2.everysec（ Per second ）
Synchronizes the instructions in the buffer to per second AOF In file , The accuracy of the data is high , Higher performance , It is recommended to use , Lost in the event of a sudden system outage 1 Seconds of data
3.no（ System control ）
The operating system controls each synchronization to AOF The period of the document , The whole process is uncontrollable

15.AOF Rewrite Preface

Look at the following questions , According to traditional thinking ,AOF Will 6 Pieces of data are written aof file , As commands are written AOF, The file will get bigger and bigger

16.AOF Introduction to rewriting

In order to solve the problem demonstrated above ,Redis Introduced AOF Rewrite mechanism to compress file volume .
AOF File rewriting is to Redis In process data is converted to write command and synchronized to new AOF Documentation process . In short, it is to convert the execution results of several commands on the same data into the instructions corresponding to the final result data for recording .

17.AOF Rewriting effect

AOF Rewriting effect
1. Reduce disk usage , Improve disk utilization
2. Improve persistence efficiency , Reduce persistent write time , Improve IO performance
3. Reduce data recovery time , Improve data recovery efficiency

18.AOF Rewriting rule

1. Data that has timed out in the process is no longer written to the file
2. Ignore invalid instructions , In process data generation is used when rewriting , So new AOF Write command to keep only the final data in the file :
Such as del key1、 hdel key2、srem key3、set key4 111、set key4 222 etc.
3. Multiple write commands for the same data are combined into one command :
Such as lpush list1 a、lpush list1 b、 lpush list1 c Can be converted to ：lpush list1 a b c.
In order to prevent client buffer overflow caused by excessive data , Yes list、set、hash、zset Other types , Each instruction can write at most 64 Elements

19.AOF Manual override mode -bgrewriteaof

bgrewriteaof

The following two demos are video screenshots ：

demonstration 1：
many times set value

After execution bgrewriteaof Output “Background append only file rewriting started” Immediately switch to “127.0.0.1：6379”, This does not mean that the rewrite has been completed , Only in the log file can we see whether the rewrite has been completed

Demo two ：
many times lpush

20.AOF Auto rewrite mode - Modify the configuration

1. It is mainly related to the following configurations :
auto-aof-rewrite-min-size size
auto-aof-rewrite-percentage percentage
2. Automatically rewrite trigger comparison parameters （ Operation instruction info
Get specific information ）:
aof_current_size ： Refers to the current situation aof How much is in the buffer , Note that this does not refer to the number of stored instructions ！
aof_base_size：
3. Automatically override trigger conditions :
Here's the picture ：

info

stay Persistence below ：

21.AOF Manual override mode -bgrewriteaof working principle

"AOF Manual override mode bgrewriteaof working principle " And bgsave Is very similar

22.AOF Rewrite process

23.RDB And AOF difference

AOF Prior to the RDB start-up

24.RDB And AOF Choice problem

1. It is very sensitive , The default... Is recommended AOF Persistence scheme
AOF Persistence strategy uses everysecond, Every second fsync once . The strategy redis It can still maintain good processing performance , When something goes wrong , At most 0-1 Seconds of data .
Be careful ： because AOF The file storage volume is large , And the recovery speed is slow
2. Data presentation phase validity , It is recommended to use RDB Persistence scheme
Data can be well done in the phase without loss （ This stage is manually maintained by developers or operation and maintenance personnel ）, And the recovery speed is faster , Phase point data recovery usually uses RDB programme
Be careful ： utilize RDB Implementing compact data persistence makes Redis It's very low , Sum up carefully ：
3. Comprehensive comparison
1）RDB And AOF Your choice is actually a trade-off , Each has its advantages and disadvantages
2） If you can't afford to lose data within a few minutes , Very sensitive to business data , choose AOF
3） If it can withstand data loss within a few minutes , And pursue the recovery speed of large data sets , choose RDB
4） Disaster recovery options RDB
5） Double insurance strategy , At the same time open RDB and AOF, After restart ,Redis priority of use AOF To recover data , Reduce the amount of lost data

25. Persistent application scenarios

None of the following redlines need persistence
Tip1： Just put the last one id+1 That's it , No persistence , Temporary storage is ok
Tip3： Refer to “ Whether the data in the cache needs to be persisted ”, Because the cached data comes from the database , So no persistence
Tip4： Shopping carts are stored in the database
Tip5： For snapping up data , Speed needs to be very fast , So you need persistence , And there are few coupons
Tip6： Be similar to Tip7, It's just temporary , Persistence is required
Tip12： If the blacklist is permanently stored , Data is stored ; If the blacklist is short-term storage ,redis Li Cun ; The white list is basically stored in the database