Redis configuration and optimization

Preface

stay web Server , High availability refers to the time when the server can be accessed normally , The measure is how long a normal service can be provided （99.9%、99.99%、99.999% wait ）.
But in Redis In context , The meaning of high availability seems to be broader , In addition to ensuring normal service （ Such as the separation of master and slave 、 Fast disaster recovery technology ）, Also need to consider the expansion of data capacity 、 Data security will not be lost .

stay Redis in , Technologies for high availability include persistence 、 Master slave copy 、 Sentinels and Cluster colony , Their functions are described below , And what kind of problems have been solved .

One 、Redis Persistence

Persistence is the simplest high availability method （ Sometimes it's not even classified as a highly available means ）, The main function is data backup , Store data on hard disk , Ensure that data is not lost due to process exit .

Persistence capability ：Redis Is a memory database , Data is stored in memory , In order to avoid server power failure and other reasons Redis Permanent loss of data after abnormal exit of the process , Need to regularly Redis In some form （ Data or commands ） Save from memory to hard disk ; The next time Redis Restart time , Using persistent files to achieve data recovery . besides , For disaster backup , You can copy persistent files to a remote location .

Redis Two ways to persist ：

1. RDB Persistence ： The principle is that Reids The database records in memory are regularly saved to disk .
2. AOF Persistence （append only file）： The principle is that Reids The operation log of is written to the file as an append , Be similar to MysQL Of binlog. because AOF Persistence is better in real time , That is, less data is lost when the process exits unexpectedly , therefore AOF It's the mainstream way of persistence , however RDB Persistence still has its place .

1.1 RDB Persistence

RDB Persistence refers to saving a snapshot of the data in the current process in memory to the hard disk within a specified time interval （ So it's also called snapshot persistence ）, Compress storage with binary , The saved file suffix is rdb; When Redis On reboot , Can read snapshot file recovery data .

RDB The trigger conditions are divided into Manual and automatic triggering

1. Manual trigger

   • save Command and bgsave Commands can be generated RDB file .
   • save Orders will block Redis Server process , until RDB Until the file is created , stay Redis During server blocking , The server cannot process any command requests .
   • bgsave The command creates a child process , The child process is responsible for creating RDB file , The parent process （ namely Redis The main process ） Then continue processing the request .
   • bgsave During command execution , Only fork Child process will block the server , And for save command , The whole process blocks the server , therefore save It's basically abandoned , Online environment should be eliminated save Use .

2. Automatic triggering

   • In auto trigger RDB Persistence ,Redis Will also choose bgsave instead of save To persist .
   • save m n： The most common case of automatic triggering is in the configuration file through save m n, Designated as m Within seconds n The next change , Will trigger bgsave.
   • perform shutdown On command , Automatic execution rdb Persistence .
   • In the master-slave replication scenario , If full replication is performed from a node , Then the master node will execute bgsave command , And will rdb The file is sent to the slave node .

vim /etc/redis/6379.conf
#--219 That's ok -- Here are three save When any condition is satisfied , Will cause bgsave Call to 
save 900 1： When the time comes 900 seconds , If redis The data happened, at least 1 Changes , execute bgsave 
save 300 10： When the time comes 300 seconds , If redis The data happened, at least 10 Changes , execute bgsave 
save 60 10000： When the time comes 60 seconds , If redis The data happened, at least 10000 Changes , execute bgsave
#--254 That's ok -- Appoint RDB file name 
dbfilename dump.rdb
#--264 That's ok -- Appoint RDB Document and AOF File directory 
dir/var/lib/redis/6379
#--242 That's ok -- Open or not RDB File compression 
rdbcompression yes

Execute the process :
（1）Redis The parent process first judges ： Is it currently being implemented save, or bgsave/bgrewriteaof Can be inherited by child processes. , If it's being executed bgsave The command returns directly to .bgsave/bgrewriteaof Cannot be executed at the same time , Mainly based on performance considerations ： Two concurrent subprocesses perform a large number of disk writes at the same time , Can cause serious performance problems .
（2） Parent process execution fork Action create subprocess , In this process, the parent process is blocked ,Redis Cannot execute any command from the client
（3） The parent process fork after ,bgsave Command return "Background saving started” Information doesn't block the parent process anymore , And can respond to other commands
（4） Subprocess creation RDB file , Generate a temporary snapshot file according to the memory snapshot of the parent process , After the completion of the original file for atomic replacement
（5） The child process sends a signal to the parent to indicate completion , Parent process update statistics

Load on startup

RDB File loading is performed automatically when the server starts , There is no special order . But because of AOF Higher priority , So when A0F On ,Redis Will load first A0F File to recover data ; Only when A0F closed , Will be in Redis Detect when the server starts RDB file , And automatically load . The server loads RDB Blocked during file , Until the load is complete .

Redis load RDB When you file , Would be right RDB Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure .

1.2 AOF Persistence

1. RDB Persistence is writing process data to a file , and AOF Persistence , Will be Redis Each write performed 、 The delete command is recorded in a separate log file , The query operation will not record ;
2. When Redis Execute again on restart AOF Command in the file to recover data .
3. And RDB comparison ,AOE Better real-time , So it has become a mainstream persistence solution .

Redis The server is turned on by default RDB, close AOF; To turn on A0F, You need to configure... In the configuration file .

vim/etc/redis/6379.conf
#--700 That's ok -- Default no, It is amended as follows yes
appendonly yes
#--704 That's ok - One designation AOF File name 
appendfilename"appendonly.aof"
#--796 That's ok -- Whether to ignore the last instruction that may have problems 
aof-load-truncated yes 
# restart redis
/etc/init.d/redis_6379 restart   

Execute the process
Because of the need to record Redis Every command written by , therefore AOF No need to trigger .

AOF The implementation process includes ：

1. Order to append （append）： take Redis The write command of is appended to the buffer aof buf;
2. File is written to （write） Synchronize with files （sync）： According to different synchronization strategies, we will aof_buf The contents of the file are synchronized to the hard disk ;
3. File rewriting （rewrite）： Rewrite regularly AOF file , Achieve the purpose of compression .

（1） Order to append （append）
Redis First append the write command to the buffer , Instead of writing directly to a file , It is mainly to avoid writing commands to the hard disk directly every time , Cause the hard disk Io Become Redis The bottleneck of the load .
The format of the command append is Redis The protocol format of the command request , It's a plain text format , Good compatibility 、 High readability 、 Easy to handle 、 It's easy to operate 、 Avoid secondary overhead and other advantages . stay AOF In file , Except for the specified database select command （ Such as select 0 For the selection 0 The database ） By Redis Added , The others are written commands sent by the client .

（2） File is written to （write） Synchronize with files （sync）
Redis Provides a variety of AOF Cache synchronization strategy , The policy involves the operating system write Functions and fsync function , The explanation is as follows ：
In order to improve the efficiency of file writing , In modern operating systems , When the user calls write Function to write data to a file , Operating systems usually store data in a memory buffer , When the buffer is filled or exceeds the specified time limit , To write the buffer data to the hard disk . This kind of operation improves the efficiency , But it also brings security issues ： If the computer goes down , Data in the memory buffer will be lost ; So the system also provides fsync、fdatasync Isochronous letter Count , It can force the operating system to write the data in the buffer to the hard disk immediately , So as to ensure the security of data .
AOF There are three synchronization methods for the cache synchronization file policy -------appendfsync always、、appendfsync no、appendfsync everysec.

1. appendfsync always： Command write aof buf Call the system immediately after fsync The operation is synchronized to A0F file ,fsync After completion, the thread returns . In this case , Every time you have a write command, you have to synchronize to A0F file , Hard disk IO Become a performance bottleneck ,Redis Only about a few hundred TPS write in , It's seriously reduced Redis Performance of ; Even with solid state drives （SSD）, It can only process tens of thousands of commands per second , And it will greatly reduce SSD Life span of .
2. appendfsync no： Command write aof buf After calling the system write operation , incorrect AOF File do fsync Sync ; Synchronization is the responsibility of the operating system , Usually the synchronization period is 30 Dirty . In this case , The timing of file synchronization is uncontrollable , And there will be a lot of data piled up in the buffer , Data security cannot be guaranteed .
3. appendfsync everysec： Command write aof_buf After calling the system write operation ,write After completion, the thread returns ;fsync The synchronization file operation is called once per second by a dedicated thread .everysec It's a compromise between the two strategies , It's a balance between performance and data security , So it is Redis Default configuration , It's also our recommended configuration .

（3） File rewriting （rewrite）
Over time ,Redis More and more write commands are executed by the server ,A0F The files will get bigger and bigger ; Too much AOF Files will not only affect the normal operation of the server , It can also cause data recovery to take too long .
File rewriting refers to periodic rewriting A0F file , Reduce A0F Volume of file . It should be noted that ,A0F Rewriting is to put Redis In process data is converted to write commands , Sync to the new A0F file ; Not to the old AOF File for any read 、 Write operation ！
Another thing to note about file rewriting is ： about AOF In terms of persistence , File rewriting is highly recommended , But it's not necessary ; Even if there is no file rewriting , Data can also be persisted and stored in Redis Import at startup ; So in some reality , Automatic file rewriting is turned off , And then through the timing task, at a certain time of the day .

The reason why file rewriting can be compressed A0F file , The reason lies in ：

1. Expired data is no longer written to the file
2. Invalid commands are no longer written to the file ： If some data is set repeatedly （set mykey v1,set mykey v2）、 Some data has been deleted （set myset v1,del myset） etc. .
3. Multiple commands can be combined into one ： Such as sadd myset v1,sadd myset v2,sadd myset v3 Can be combined into sadd myset vl v2 v3.

From the above, we can see that , Because after rewriting AOF There are fewer orders to execute , File rewriting can reduce the space occupied by the file , It can also speed up recovery .

Trigger of file rewriting , It can be divided into manual trigger and automatic trigger ：

1. Manual trigger ： Call directly bgrewriteaof command , The execution of the order is related to bgsave Some similar ： All are fork The subprocess does the specific work , And only in fork Time blocking .
2. Automatic triggering ： By setting auto-aof-rewrite-min-size Options and auto-aof-rewrite-percentage Option to automate BGREWRITEAOF.
   Only when auto-aof-rewrite-min-size and auto-aof-rewrite-percentage When both options are met , Will automatically trigger AOF rewrite , namely bgrewriteaof operation .

vim/etc/redis/6379.conf
#--729--
auto-aof-rewrite-percentage 100
# At present AOF file size （ namely aof_current_size） Is the last time the log was rewritten AOF file size （aof_base_size） At twice , happen BGREWRITEAOF operation 
auto-aof-rewrite-min-size 64mb
# At present AOF File execution BGREWRITEAOF The minimum value of the command , Avoid starting Reids Due to the small file size, frequent BGREWRITEAOF

About the file rewriting process , There are two points to note ：（1） Rewriting by the parent process fork The subprocess goes on ;（2） During rewrite Redis Write commands executed , Need to be added to the new AOF In file , So Redis Introduced aof_rewrite_buf cache .

The process of file rewriting is as follows ：
（1）Redis The parent process first determines whether there is currently executing bgsave/bgrewriteaof Can be inherited by child processes. , If there is one bgrewriteaof The command returns directly to , If there is bgsave Orders wait bgsave Execute after the execution is completed .
（2） Parent process execution fork Action create subprocess , In this process, the parent process is blocked .
（3.1） The parent process fork after ,bgrewriteaof Command return "Background append only file rewrite started" Information doesn't block the parent process anymore , And can respond to other commands .Redis All write commands for are still written to AOF buffer , And according to appendfsync Policy synced to hard disk , Keep the original AOF The right mechanism .
（3.2） because fork The operation uses copy on write technology , Child processes can only share fork Memory data during operation . Because the parent process is still responding to commands , therefore Redis Use AOF Rewrite buffer （aof_rewrite_buf） Save this data , Prevent new AOF This data is lost during file generation . in other words ,bgrewriteaof During execution ,Redis At the same time, the write command of aof_buf and aof_rewirte_buf Two buffers .
（4） Sub process according to memory snapshot , Write to the new... According to the command merge rule AOF file .
（5.1） The subprocess writes a new A0F After the document , Signal the parent process , Parent process update statistics , The details can be obtained through info persistence see .
（5.2） The parent process put AOF The data in the rewrite buffer is written to the new AOF file , This guarantees new AOF The database state saved by the file is consistent with the current state of the server .
（5.3） Use the new A0F File replace old file , complete AOF rewrite .

Load on startup

When A0F On ,Redis When it starts, it takes precedence to load AOF File to recover data ; Only when A0F closed , Will load RDB File recovery data .
When AOF Turn on , but A0F When the file does not exist , Even if RDB The file will not load if it exists .
Redis load AOF When you file , Would be right AOF Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure . But if it is A0F The end of the file is incomplete （ The end of the file is not complete when the machine goes down suddenly ）, And aof-load-truncated Parameter on , A warning will be output in the log ,Redis Ignore AOF End of file , Successful launch .aof-load-truncated The parameter is on by default .

1.3 RDB and AOF Advantages and disadvantages

1. RDB Persistence

   • advantage ：
     RDB The files are compact , Small volume , Network transmission is fast , Suitable for full replication ; Recovery speed ratio AOF Much faster . Of course , And A0F comparison ,RDB One of the most important advantages is that the impact on performance is relatively small .
   • shortcoming ：
     • RDB The fatal disadvantage of the file is that the persistence mode of the data snapshot determines that the real-time persistence cannot be achieved , And today, data is more and more important , The massive loss of data is often unacceptable , therefore AOF Persistence becomes the mainstream . Besides ,RDB The file needs to be in a specific format , Compatibility is poor （ Like the old version Redis Not compatible with the new version of RDB file ）.
     • about RDB Persistence , On the one hand is bgsave It's going on fork In operation Redis The main process will block , On the other hand , Sub process writing data to the hard disk will also bring Io pressure .

2. AOF Persistence

   • And RDB Persistence corresponds to ,AOF The advantage of this is that it supports second level persistence 、 Compatibility is good. , The disadvantage is that the files are large 、 Slow recovery 、 Great impact on performance .
   • about A0F Persistence , The frequency of writing data to the hard disk is greatly increased （everysec Second level under strategy ）,IO More stressful , It may even cause A0F Additional blocking problem .
   • A0F Document rewriting and RDB Of bgsave similar , There will be fork The blocking and the subprocess's IO The pressure problem . relatively speaking , because AOF Write data to the hard disk more often , So right. Redis The performance of the main process will be affected more .

Two 、Redis Performance management

info memory： stay Redis Use this command in Redis View of memory usage 
--------------------- Memory fragmentation rate ----------------------
mem_fragmentation_ratio： Memory fragmentation rate .
mem_fragmentation_ratio=used_memory_rss/used_memory 
used_memory_rss： yes Redis Memory requested from the operating system .
used memory： yes Redis Memory occupied by data in .
used_memory_peak:redis Peak memory usage 

2.1 How memory fragmentation occurs

1. Redis It has its own memory manager , In order to improve the efficiency of memory use , To manage the application and release of memory .
2. Redis When the value in is deleted , Did not release memory directly , Give it back to the operating system , It was handed over Redis There is a memory manager inside .
3. Redis When applying for memory in , First, check whether there is enough memory available in your memory manager .
4. Redis This mechanism of , Improved memory usage , But it will make Redis Some of them are not in use , But not free memory , Caused memory fragmentation .

notes ： Track memory fragmentation rate for understanding Redis The resource performance of an instance is very important .

1. The memory fragmentation rate is 1 To 1.5 Between is normal , This value indicates that the memory fragmentation rate is relatively low , Also explain Redis No memory swap occurred .
2. The memory fragmentation rate exceeds 1.5, explain Redis It consumes the physical memory 150%, among 50% It's the memory fragmentation rate .
3. The memory fragmentation rate is lower than 1 Of , explain Redis Memory allocation exceeds physical memory , The operating system is swapping memory . Need to increase available physical memory or reduce Redis Memory footprint .

2.2 Solve the problem of high fragmentation rate

1. If your Redis The version is 4.0 Following , Need to be in redis-cli Enter... On the tool shutdown save command , Give Way Redis The database performs a save operation and closes Redis service , Restart the server again .Redis After the server restarts ,Redis Will return unused memory to the operating system , The fragmentation rate will come down .
2. Redis4.0 Version start , It can be done without restarting , Online defragmentation of memory .

config set activedefrag yes
# Automatic debris removal , The memory will clean up automatically .
memory purge   
# Manual debris removal 

2.3 Memory usage

redis The memory usage of the instance exceeds the maximum available memory , The operating system will start memory and memory swap Space exchange .

Ways to avoid memory swapping ：

1. Select the installation for the cache data size Redis example
2. Use as much as possible Hash Data structure storage
3. Set up key The expiration time of

Memory cleaning strategy , Ensure reasonable distribution redis Limited memory resources .

When the set maximum threshold is reached , You need to select one key Recycling strategy for , By default, the recycling policy is to prohibit deletion .

Modify... In configuration file maxmemory-policy Property value ：

vim/etc/redis/6379.conf
#--598--
maxmemory-policy noenviction

Recovery strategy ：

1. volatile-lru： Use LRU The algorithm eliminates the data from the data set that has set the expiration time （ Remove the least recently used key, Set... For TTL Of key）

• volatile-ttl： Select the data that is about to expire from the data set with expiration time （ Remove recently expired key）
• volatile-random： Randomly select data from the data set with expiration time set （ In setting up the TTL Of key Randomly remove ）
• allkeys-lru： Use LRU Algorithms eliminate data from all data sets （ Remove the least used key, For all key）
• allkeys-random： Select any data from the data set （ Remove randomly key）
• noenviction： Ban data obsolescence （ Don't delete until it's full ）

So it's set up

To sum up

1. RDB and AOF The process of
2. RDB and AOF Advantages and disadvantages
3. AOF Cache synchronization strategy