Interviewer: you said you are proficient in redis. Have you seen the persistent configuration?

We have already introduced

Redis

Commands for five data types

And

The basic configuration of the configuration file

, Today, let's uncover it from two aspects: Theory and configuration

Redis

A lasting veil of mystery .

The so-called persistence can be simply understood as the process of saving the data in memory to the hard disk for storage . The persistent data can still be accessed after system restart or downtime , Ensure the security of data .

Redis

There are two persistence schemes , One is snapshot （

SNAPSHOTTING

）, abbreviation

RDB

; One is append only mode （

APPEND ONLY MODE

）, be called AOF. Next, let's take a look at their use and precautions .

RDB

RDB

by

Redis DataBase

Abbreviation , yes  

Redis

  Default persistence scheme . It can snapshot memory data sets within a specified time interval （

snapshot

） Write to disk , Snapshot files will be restored （ 

dump.rdb

 ） Read back memory .

Let's start with the... In the configuration file

SNAPSHOTTING

:

The configuration file

save 

<seconds> <changes>

In the given

Number of seconds

Inside , If you execute on the database

Write operand

Reach the set value , Then synchronize the data to the data file . Support multiple conditions ,

Redis

Three conditions are provided in the default configuration file ：

save 900 1 //900s There are 1 A change
save 300 10 //300s There are 10 A change
save 60 10000 //60s There are 10000 Changes


Be careful

： If you don't want to use

RDB

programme , You can put  

save "" 

The comment opens , Note out the three above .

stop-writes-on-bgsave-error yes

When

bgsave

When something goes wrong ,

Redis

Whether to stop executing the write command ;

• If
  yes
  , When there is a problem with the hard disk ,
  Redis
  Will stop accepting writes , So we can find out in time , Avoid massive loss of data ;

• If
  no
  , be
  Redis
  ignore
  bgsave
  Continue to execute the write command .

If it has been set to

Redis

Proper monitoring and persistence of the server , That is, other means are used to discover and control data integrity , You may want to disable this feature , So that even on disk 、 When there is a problem with permissions ,

Redis

Still working .

Be careful

： If the background save process starts working again ,

Redis

Will automatically allow writing again .

rdbcompression yes

Specifies whether to store to the local database

Compress

（

Redis

use

LZF

Compress ） data , The default is

yes

. If in order to save money

CPU

Time , This option can be turned off , But it will cause the database file to become huge .

rdbchecksum yes

from

RDB

edition

5

Start , After storing the snapshot , You can also use

CRC64

Algorithm to check data ,

CRC64

The verification is placed at the end of the file . After opening , Save and load

RDB

Files will be added about

10%

Performance consumption of , If you want to get the maximum performance improvement , You can turn it off .

Ban

The checksum

Created

RDB

The check sum of the file is zero , This will tell the loading code to skip the check .

dbfilename dump.rdb

Specifies the local database file name , Automatically load in after restart

Memory

, Do it manually

save

  The order will take effect immediately .

Big pit, please pay attention to

：

flushall

、

shutdown

All commands will be cleared and submitted to

dump.rdb

dir ./

Specifies the local database hosting directory .

theory

Operation mode

• When  
  Redis
    Need to save
  dump.rdb
  When you file , It calls system functions
  fork()
  , Create a subprocess （ Completely consistent with the main process ）;

• The child process writes the data set to a temporary file
  RDB
  in ;

• When the subprocess completes the new  
  RDB
    When the file is written ,
  Redis
    New use  
  RDB
    Replace the original file with  
  RDB
    file , And delete the old  
  RDB
    file .

This way of working makes  

Redis

  Can be copied from write time （

copy-on-write

） Benefit from the mechanism .

How to trigger RDB snapshot

• The default snapshot configuration in the configuration file ;

• command
  save
  （ Blocking ,  Just save the snapshot , Other waiting ） Or is it
  bgsave
  （ asynchronous ） command , Snapshots can also respond to client commands ;

• perform
  flushall
    command , Clear all data in the database , It doesn't make much sense ;

• perform
  shutdown
    command , Make sure that the server is shut down properly without losing any data , It doesn't make much sense .

adopt RDB File recovery data

In actual development , Generally, the physical hard disk is damaged , So we will choose backup

dump.rdb

file . Will back up

dump.rdb

  File copy to

redis

Of the installation directory

bin

Under the table of contents , restart

redis

The service can be .

advantage

• RDB
  It's a very compact file , Very suitable for data set backup ;

• RDB
  It's a compact single file , It's easy to transfer to another remote data center or Amazon's S3（ It could be encrypted ）, Ideal for disaster recovery ;

• Redis
  The main process of is not running
  I/O
  operation , Ensures extremely high performance ;

• Suitable for large-scale data recovery , If the requirements for data integrity and consistency are not high ,
  RDB
  Than
  AOF
  It's more efficient .

shortcoming

• stay
  Redis
  In case of unexpected downtime , You may lose a few minutes of data ;

• RDB
    Need to often
  fork
  Subprocess to save the data set to the hard disk , When the data set is large ,
  fork
  The process is very time consuming , May lead to
  Redis
  Can't respond to client requests in milliseconds . If the data set is huge and
  CPU
  When the performance is not very good , This situation will continue 1 second ;
  AOF
  Also needed
  fork
  , However, you can adjust the frequency of rewriting log files to improve the durability of data sets .

AOF

In order to solve

RDB

The problem of losing too much data during downtime , from

1.1

  Version start ,

Redis

Added a kind of

durable

Persistence of ：

AOF

.

AOF

yes

Append Only File

Abbreviation , Not on by default .

AOF

Log every write operation in the form of a log , You can only append files, but you can't overwrite files , These commands are reexecuted when the server restarts to restore the original data .

Let's take another look at... In the configuration file

APPEND ONLY MODE

:

The configuration file

appendonly no

The default is off , Change it to

yes

Turn on persistence .

AOF

and

RDB

Can be enabled at the same time without problems .

appendfilename "appendonly.aof"

File default name , Start to create . load

Precede

dump.rdb

file

appendfsync

Synchronization strategies ： System function

fsync()

  Tell the operating system to actually write data on the disk .

Redis

Support three different modes

appendfsync always // Every time there is a data change, it will be recorded to the disk immediately , Poor performance but good data integrity
appendfsync everysec // Default recommendation , Asynchronous operations , Record every second , If it goes down , Yes 1 Data loss in seconds
appendfsync no // Out of sync , Refresh data only when the operating system needs it


To understand the next configuration , First of all “ Log to rewrite ” Principle ：

rewrite

because

AOF

The method is to append the command to the end of the file , So as the number of write commands increases ,

AOF

The size of the file will become larger and larger . To avoid this , New rewrite mechanism ： You can do this without interrupting the service client , Yes

AOF

Document reconstruction （

rebuild

）.

Override trigger ：

  Through execution

bgrewriteaof

command , You can generate a new

AOF

file , This file contains the data needed to rebuild the current dataset

least

command .

Redis 2.2

This command needs to be executed manually ,

Redis 2.4

Can be triggered automatically by modifying the configuration file （ The configuration involves ）.

Rewriting principle ：

• Redis
    Execute system functions
  fork()
   , Create a subprocess （ Completely consistent with the main process ）;

• The subprocess begins to change  
  AOF
    The contents of the file are written to a temporary file ;

• For all newly executed write commands , The parent process accumulates them in a memory cache , While adding these changes to the existing  
  AOF
    End of file , So even if there's a stoppage in the middle of the rewrite , The existing  
  AOF
    Files are also safe ;

• When the subprocess finishes rewriting , It sends a signal to the parent process , After receiving the signal, the parent process , Append all data in the memory cache to the new  
  AOF
    End of file .

• Redis
    Atomically replace old files with new ones , After that, all commands will be added directly to the new  
  AOF
  End of file .

no-appendfsync-on-rewrite no

When we execute the main process at the same time

Write operations

And subprocess

rewrite

In operation , Both operate on disks , Rewriting often involves a lot of disk operations , This will cause the main process to write

aof

Blocking occurs when the file is .

To solve this problem ,

no-appendfsync-on-rewrite

The parameters come out .

• If the parameter is set to
  no
  , It's the safest way , No loss of data , But endure the problem of congestion ;

• If set to
  yes
  , This is equivalent to putting
  appendfsync
  Set to
  no
  , This indicates that no disk operation was performed , Just write to the buffer . So this doesn't cause congestion （ Because there are no competing disks ）, But if this time
  redis
  Hang up , You lose data . How much data is lost ？ stay
  linux
  Under the default settings of the operating system , The most you can lose 30s The data of .

therefore , If the application system can't stand the delay , And can tolerate a small amount of data loss , Is set to

yes

; If the application system can't tolerate data loss , Is set to

no

.

auto-aof-rewrite-percentage 100

Rewrite percentage , The default is... After last rewriting

aof

Double the file size .

auto-aof-rewrite-min-size 64mb

The minimum value of the override trigger ：64mb.

according to

auto-aof-rewrite-min-size

and

auto-aof-rewrite-percentage

Parameter determination of automatic trigger time .

Redis

It will record the last time it was rewritten

AOF

size , The default configuration is when

AOF

The file size is last

rewrite

Double the size and the file is larger than

64M

Trigger when .

Large Internet companies are generally

3G

start

aof-load-truncated yes

When

AOF

When the file is truncated , namely

AOF

The last command in the file is incomplete , If you start at this point

Redis

, Will

AOF

Data is loaded back into memory , At this point, a problem occurs .

• yes： Load a truncated
  AOF
  ,
  Redis
  The server started issuing logs , Notify the user of this event ;

• no： The server will abort with an error , Refuse to start .

When we learned

AOF

When the file reports an error , You can use the following methods to fix the wrong  

AOF

  file ：

• For the existing  
  AOF
  File to create a backup ;

• Use  
  Redis
    Incidental  
  redis-check-aof
    command , To the original
  AOF
  File for repair ;

• redis-check-aof –fix

• redis-check-aof --fix appendonly.aof
    Repair order , Kill all non-conforming grammar

• （
  Optional
  ） Use  
  diff -u
    Compare the repaired  
  AOF 
  Documents and original  
  AOF
    Backup of files , Look at the differences between the two files ;

• restart  
  Redis
  The server , Wait for the server to load the repaired  
  AOF 
  file , And data recovery .

aof-use-rdb-preamble yes

In rewriting

AOF

When you file ,

Redis

In the

AOF

Use... In the document

RDB

Leading , To speed up rewriting and recovery . When this option is enabled , Rewrite the

AOF

The file consists of two different sections ：

RDB file

、

AOF tail

load

Redis

when , identifies

AOF

Document to

Redis

Start of string , And load prefixed

RDB

file , Then continue loading

AOF

The tail .

theory

advantage

• Higher data integrity and consistency ,
  AOF
  By using different strategies , At most 1 Second data ;

• AOF
  A file is a log file that only appends , No need to write
  seek
  ;

• Redis
  Can be in  
  AOF 
  When the file size becomes too large , Automatically in the background  
  AOF
    Rewrite , Rewriting is absolutely safe ;

• AOF
  The write operation of the file record is performed in
  Redis
  The format of the protocol is saved , Easy to read , Easy to analyze documents ;

shortcoming

• For the same dataset ,
  AOF
  The volume of the file is usually larger than
  RDB
  Volume of file ;

• According to the  
  fsync
    Strategy ,
  AOF
  May be slower than
  RDB
   .

In general , Per second  

fsync

  Performance is still very high , Shut down  

fsync

  It can make  

AOF

  Speed and  

RDB

  As fast as ,  Even under high load .  But when dealing with large write loads ,

RDB

  More guaranteed maximum delay time （

latency

）.

Comparison and summary

How to choose which persistence method to use ？

Generally speaking , If you want to achieve enough  

PostgreSQL

  Data security of , Two persistence functions should be used at the same time .

If you are very concerned about data , But it can still withstand data loss within a few minutes , Then you can only use  

RDB

  Persistence .

because AOF Persistence is better in real time , That is, less data is lost when the process exits unexpectedly , therefore

AOF

Is currently the

Main stream

Persistence of .

Many users only use

AOF

Persistence , But we don't recommend it ： Because of timing generation  

RDB

  snapshot （

snapshot

） Very convenient for database backup , also  

RDB

  Data set recovery is also faster than  

AOF

  Fast recovery .

AOF and RDB The interaction between

The version number is greater than or equal to  

2.4

  Of  

Redis

  in ,

BGSAVE

  During execution , Can't execute  

BGREWRITEAOF

 . On the other hand , stay  

BGREWRITEAOF

  During execution , You can't do it  

BGSAVE

. This can prevent two  

Redis

  The background process does a lot of work on the disk at the same time

 I/O

  operation .

If  

BGSAVE

  Being implemented , And the user calls  

BGREWRITEAOF

  command , Then the server will reply to the user with a  

OK

  state ,  And tell the user

BGREWRITEAOF

  Has been scheduled to execute ： once  

BGSAVE

  completion of enforcement ,

BGREWRITEAOF 

It will officially start .

When  

Redis

  Startup time , If  

RDB

Persistence and  

AOF

  Persistence is turned on ,  Then the program will give priority to  

AOF

  File to recover the dataset , because  

AOF 

The data stored in the file is usually the most complete .

Backup redis data

• Create a recurring task （
  cron job
  ）, There will be one... Per hour  
  RDB
    File backup to a folder , And every day there will be one  
  RDB
    File backup to another folder ;

• Make sure that the backup of the snapshot has the corresponding date and time information , Every time you execute a regular task script , Use  
  find
    Command to delete expired snapshots ;

• At least once a day , take  
  RDB
    Back up outside your data center , Or at least back it up to you  
  Redis
    Outside the physical machine of the server .

Performance Suggestions

In practice , because

RDB

The documents are for backup purposes only , The advice is only in

slave

On persistence

RDB

file , And it just needs 15 One minute backup is enough , Only keep

save 900 1

This rule .

If open

AOF

, The advantage is that in the worst case, it will only be lost no more than 2 Second data , The startup script is simple

load

Their own

AOF

Just file it . The first is to bring about continuous

IO

, Two is

AOF rewrite 

At the end of

rewrite

The blocking caused by writing new data to new files is almost inevitable .

As long as the hard disk is licensed , We should try to minimize

AOF rewrite

The frequency of ,

AOF

Overridden base size default

64M

Is too small , It can be set to

5G

above . The default size exceeds the original size 100% When overridden, it can be changed to the appropriate value .

If you don't turn it on

AOF

, Only by

Master-Slave Replication

High availability can also be achieved . Can save a lot of

IO

, And less

rewrite

The fluctuation of the system caused by . The price is if

Master/Slave 

At the same time , It will lose more than ten minutes of data , There are also two startup scripts to compare

Master/Slave

Medium

RDB

file , Load the newer one .

o  Q  Will continue to update  java  Practical articles , If you have a different opinion or a better one  idea, Welcome to contact ah  Q.

【 o  Q  Say code 】, Notable official account

The style of the article is changeable , The pictures are easy to understand , The story is vivid and interesting , Let's talk about technology ！