A complete tutorial for getting started with redis: AOF persistence

        AOF（append only file） Persistence ： Each write is logged as a separate log ,
Reexecute on reboot AOF The command in the file restores the data .AOF Main role of
It solves the real-time problem of data persistence , So far Redis The mainstream way of persistence . understand
Master well AOF Persistence mechanism is very helpful for us to balance data security and performance .

5.2.1　 Use AOF
Turn on AOF Function requires configuration ：appendonly yes, Not on by default .AOF file name
adopt appendfilename Configuration settings , The default filename is appendonly.aof. Save path is the same as
RDB Consistent persistence , adopt dir Configuration assignment .AOF The workflow operation of ： Command write
（append）、 File synchronization （sync）、 File rewriting （rewrite）、 Restart loading
（load）, Pictured 5-2 Shown . 

The process is as follows ：
1） All write commands are appended to aof_buf（ buffer ） in .
2）AOF The buffer synchronizes the hard disk according to the corresponding policy .
3） With AOF The files are getting bigger , It needs to be done regularly AOF File rewriting , Compression reached
Purpose .
4） When Redis When the server restarts , Can be loaded AOF File for data recovery .
understand AOF After the workflow , The following is a detailed introduction to each step . 

5.2.2　 Command write
AOF The content written by the command is directly in the format of text protocol . for example set hello world This life
Make , stay AOF The buffer will be appended with the following text ：
*3\r\n$3\r\nset\r\n$5\r\nhello\r\n$5\r\nworld\r\n
Redis See 4.1 Client protocol section , No more details here , Below
About AOF Two doubts ：
1）AOF Why use text protocol format directly ？ Possible reasons are as follows ：
· Text protocol has good compatibility .
· Turn on AOF after , All write commands contain append operations , Directly in the form of agreement , avoid
Avoid secondary processing overhead .
· Text protocols are readable , Easy to modify and process directly .
2）AOF Why add a command to aof_buf in ？Redis Responding to commands with a single thread , Such as
Every time I write AOF File commands are all appended directly to the hard disk , Then the performance depends entirely on the current hard disk negative
load . Write buffer first aof_buf in , There is another benefit ,Redis Multiple buffers can be provided
Strategy of synchronizing hard disk , Balance performance and security .

5.2.3　 File synchronization
Redis Provides a variety of AOF Buffer synchronization file policy , By the parameter appendfsync control ,
The meanings of different values are shown in table 5-1 Shown .

surface 5-1 AOF Buffer synchronization file policy

  system call write and fsync explain ：
·write Operation will trigger delay write （delayed write） Mechanism .Linux Provide page caching in the kernel
The flush area is used to improve the performance of the hard disk IO performance .write The operation directly returns... After writing to the system buffer . Sync
Hard disk operation depends on the system scheduling mechanism , for example ： Buffer page space is full or reaches a certain time
period . Before synchronizing files , If the system fails at this time , Data in the buffer will be lost .
·fsync For single file operations （ such as AOF file ）, Do force hard disk synchronization ,fsync take
Blocks until writing to the hard disk is complete and returns , Data persistence is guaranteed .
except write、fsync,Linux It also provides sync、fdatasync operation , Specifically API Description reference
see ：http://linux.die.net/man/2/write,http://linux.die.net/man/2/fsync,http://linux.die.net/man/2/sync
· Configure to always when , Synchronize every write AOF file , In a normal SATA Hard disk
On ,Redis Only about a few hundred TPS write in , Obviously with Redis High performance is the opposite ,
Configuration not recommended .
· Configure to no, Due to every synchronization of the operating system AOF Uncontrollable period of file , And will add
Large amount of data per synchronization of the hard disk , While improving performance , But data security cannot be guaranteed .
· Configure to everysec, Is the recommended synchronization strategy , Is also the default configuration , Achieve both performance and performance
Data security . In theory, it can only be lost when the system suddenly goes down 1 Second data .（ Strictly
At most 1 The second data is inaccurate ,5.3 The section will make a specific introduction to .）

5.2.4　 Rewrite mechanism
As commands are written AOF, The file will get bigger and bigger , To solve this problem ,Redis
introduce AOF Rewrite mechanism to compress file volume .AOF File rewriting is to Redis In process data transfer
Convert to write command and synchronize to new AOF Documentation process .
The rewritten AOF Why files can be smaller ？ There are the following reasons ：
1） Data that has timed out in the process is no longer written to the file .
2） old AOF File contains invalid command , Such as del key1、hdel key2、srem keys、set
a111、set a222 etc. . Rewriting is directly generated using in-process data , So new AOF The document only guarantees
Leave the write command of the final data .
3） Multiple write commands can be combined into one , Such as ：lpush list a、lpush list b、lpush list
c Can be converted to ：lpush list a b c. To prevent a single command from overflowing the client buffer
Out , about list、set、hash、zset Equal type operation , With 64 Divided into multiple elements .
AOF Rewriting reduces file footprint , besides , Another purpose is ： Smaller AOF
Files can be made faster by Redis load .
AOF The rewriting process can be triggered manually and automatically ：
· Manual trigger ： Call directly bgrewriteaof command .
· Automatic triggering ： according to auto-aof-rewrite-min-size and auto-aof-rewrite-percentage ginseng
The number determines the automatic trigger timing .
·auto-aof-rewrite-min-size： Said to run AOF Minimum file size on rewrite , Default
by 64MB.
·auto-aof-rewrite-percentage： On behalf of the current AOF File space
（aof_current_size） And after the last rewrite AOF File space （aof_base_size） than
value .
Automatic trigger time =aof_current_size>auto-aof-rewrite-min-
size&&（aof_current_size-aof_base_size）/aof_base_size>=auto-aof-rewrite-
percentage
among aof_current_size and aof_base_size Can be in info Persistence Check the statistical information
see .
When triggered AOF When rewriting , What has been done internally ？ The following is combined with the figure 5-3 Introduce its operation
technological process .

 

Process description ：
1） perform AOF Rewrite request .
If the current process is executing AOF rewrite , The request does not execute and returns the following response ：
ERR Background append only file rewriting already in progress
If the current process is executing bgsave operation , The rewrite command is delayed until bgsave After that
perform , Return the following response ：
Background append only file rewriting scheduled
2） Parent process execution fork Create child process , The cost is equal to bgsave The process .
3.1） The main process fork After the operation is completed , Continue to respond to other orders . All modification commands are still written
Enter into AOF Buffer and according to appendfsync Policy synced to hard disk , Keep the original AOF The mechanism is right
sex .
3.2） because fork Operations use write time replication technology , Child processes can only share fork Internal during operation
Save the data . Because the parent process is still responding to commands ,Redis Use “AOF Rewrite buffer ” Save this
New data , Prevent new AOF This data is lost during file generation .
4） Sub process according to memory snapshot , Write to the new... According to the command merge rule AOF file . Every time
The amount of data written to the hard disk in one batch is configured by aof-rewrite-incremental-fsync control , The default is
32MB, Prevent the hard disk from blocking due to too much data in a single brush .
5.1） new AOF When the file is written , The child process signals the parent process , Parent process update
Statistics , Specific view info persistence Under the aof_* Relevant statistics .
5.2） The parent process put AOF The data in the rewrite buffer is written to the new AOF file .
5.3） Using the new AOF File replace old file , complete AOF rewrite .

5.2.5　 Restart loading
AOF and RDB Files can be used for data recovery when the server restarts . Pictured 5-4 Shown ,
Express Redis Persistent file loading process .

chart 5-4 Redis Persistent file loading process

 

Process description ：
1）AOF Persistence is on and exists AOF When you file , Priority load AOF file , Print as follows
journal ：
* DB loaded from append only file: 5.841 seconds
2）AOF Close or AOF When the file does not exist , load RDB file , Print the following log ：
* DB loaded from disk: 5.586 seconds
3） load AOF/RDB After the document is successful ,Redis Successful launch .
4）AOF/RDB When there is an error in the file ,Redis Failed to start and print error message . 

5.2.6　 File check
Load corrupted AOF File will refuse to start , And print the following log ：
# Bad file format reading the append only file: make a backup of your AOF file,
then use ./redis-check-aof --fix <filename>
Operation and maintenance tips
For malformed AOF file , Backup first , Then use redis-check-aof--fix life
Make repairs , Use after repair diff-u Differences in comparative data , Find the missing data , Some can be
Complete with manual modification .
AOF The end of the file may be incomplete , For example, the sudden power failure of the machine leads to AOF The tail
Incomplete writing of file command .Redis For us aof-load-truncated Configured to be compatible with this situation
condition , Default on . load AOF when , When this problem is encountered, it will ignore and continue to start , Print at the same time
The following warning log ：
# !!! Warning: short read while loading the AOF file !!!
# !!! Truncating the AOF at offset 397856725 !!!
# AOF loaded anyway because aof-load-truncated is enabled