Readis configuration and optimization of NoSQL (final chapter)

Catalog

Preface ： stay Redis In context , The meaning of high availability is broader , In addition to ensuring normal service ( Such as the separation of master and slave 、 Fast disaster recovery technology ), Also need to consider the expansion of data capacity , Data security will not be lost

One ：Redis High availability

stay web Server , High availability refers to the time when the server can be accessed normally , The measure is how long a normal service can be provided (99.9%、 99.99%、 99.999% wait )
But in Redis In context , The meaning of high availability seems to be broader , In addition to ensuring normal service ( Such as the separation of master and slave 、 Fast disaster recovery technology ), Also need to consider the expansion of data capacity , Data security will not be lost
stay Redis in , Technologies for high availability include persistence 、 Master slave copy 、 Sentinels and groups , It works as follows ：
Persistence ： Persistence is the simplest high availability method ( Sometimes it's not even classified as a highly available means ), The main function is data backup , Store data on hard disk , Ensure that data is not lost due to process exit
Master slave copy ： Master slave replication is highly available Redis The basis of , Sentinels and clusters are highly available based on master-slave replication . Master-slave replication mainly realizes multi machine backup of data , And load balancing and simple fault recovery for read operations . defects : Failure recovery cannot be automated ; Write operations are not load balanced ; Storage capacity is limited by a single machine
sentry ： On the basis of master-slave replication , Sentinels achieve automated recovery from malfunctions . defects : Write operations are not load balanced ; Storage capacity is limited by a single machine
colony ： By clustering ,Redis It solves the problem that write operation cannot be load balanced , And storage capacity is limited by single machine , A relatively complete high availability scheme has been realized

---

Two ：Redis Persistence

2.1 Persistence capability

Redis Is a memory database , Data is stored in memory , In order to avoid server power failure and other reasons Redis Permanent loss of data after abnormal exit of the process , Need to regularly Redis In some form ( Data or commands ) Save from memory to hard disk ; The next time Redis Restart time , Using persistent files to achieve data recovery .
besides , For disaster backup , You can copy persistent files to a remote location

2.2 Two ways of persistence

RDB Persistence (Redis DataBase)： The principle is that Reids The database records in memory are regularly saved to disk
AOF Persistence (append only file)： The principle is that Reids The operation log of is written to the file as an append , Be similar to MySQL Of binlog
summary ： because AOF Persistence is better in real time , That is, less data is lost when the process exits unexpectedly , therefore AOF It's the mainstream way of persistence , however RDB Persistence still has its place

---

3、 ... and ：RDB Persistence

RDB Persistence refers to saving a snapshot of the data in the current process in memory to the hard disk within a specified time interval ( So it's also called snapshot persistence ), Compress storage with binary , The saved file suffix is rdb; When Redis On reboot , Can read snapshot file recovery data

3.1 The trigger condition

RDB The trigger of persistence is divided into ： Manual and automatic triggering Two kinds of

3.1.1 Manual trigger

save Command and bgsave Commands can be generated RDB file .
save Orders will Blocking Redis Server process , until RDB Until the file is created , stay Redis During server blocking , The server cannot process any command requests .
bgsave The command creates a child process , The child process is responsible for creating RDB file , The parent process ( namely Redis The main process ) Then continue processing the request .
bgsave During command execution , Only fork Child process will block the server , And for save command , The whole process blocks the server , therefore save It's basically abandoned , Online environment should be eliminated save Use ！！！

3.1.2 Automatic triggering

In auto trigger RDB Persistence ,Redis also Will choose bgsave instead of save To persist

save m n

The most common case of automatic triggering is in the configuration file through save m n, Designated as m Within seconds n The next change , Will trigger bgsave

vim /etc/redis/6379.conf-----219 That's ok -- Here are three save When any condition is satisfied , Will cause bgsave Call to save 900 1 : When the time comes 900 seconds , If redis The data happened, at least 1 Changes , execute bgsavesave 300 10 : When the time comes 300 seconds ,  If redis The data happened, at least 10 Changes , execute bgsavesave 60 10000 : When the time comes 60 seconds , If redis The data happened, at least 10000 Changes ,  execute bgsave-----242 That's ok -- Open or not RDB File compression rdbcompression yes-----254 That's ok -- Appoint RDB file name dbfilename dump.rdb-----264 That's ok -- Appoint RDB Document and AOF File directory dir /var/lib/redis/6379

3.1.3 Other automatic trigger mechanisms

except savemn outside , There are other situations that trigger bgsave
In the master-slave replication scenario , If full replication is performed from a node , Then the master node will execute bgsave command , And will rdb The file is sent to the slave node .
perform shutdown On command , Automatic execution rdb Persistence

3.2 Execute the process

（1）Redis The parent process first judges : Is it currently being implemented save, or bgsave/bgrewriteaof Can be inherited by child processes. , If it's being executed bgsave The command returns directly to .bgsave/bgrewriteaof Cannot be executed at the same time , Mainly based on performance considerations : Two concurrent subprocesses perform a large number of disk writes at the same time , Can cause serious performance problems .
（2） Parent process execution fork Action create subprocess , In this process, the parent process is blocked ,Redis Cannot execute any command from the client
（3） The parent process fork after ,bgsave Command return ”Background saving started" Information doesn't block the parent process anymore , And can respond to other commands
（4） Subprocess creation RDB file , Generate a temporary snapshot file according to the memory snapshot of the parent process , After the completion of the original file for atomic replacement
（5） The child process sends a signal to the parent to indicate completion , Parent process update statistics

3.3 Load on startup

RDB File loading is performed automatically when the server starts , There is no special order . But because of A0F Higher priority , So when AOF On ,Redis Will load first AOF File to recover data ; Only when A0F closed , Will be in Redis Detect when the server starts RDB file , And automatically load . The server loads RDB Blocked during file , Until the load is complete .
Redis load RDB When you file , Would be right RDB Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure .

---

Four ：AOF Persistence

RDB Persistence is writing process data to a file , and AOF Persistence , Will be Redis Each write performed 、 The delete command is recorded in a separate log file , The query operation will not record ; When Redis Execute again on restart AOF Command in the file to recover data .
And RDB comparison ,AOF Better real-time , So it has become a mainstream persistence solution

4.1 Turn on AOF

Redis The server is turned on by default RDB, close AOF Of , To turn on AOF, You need to configure... In the configuration file

vim /etc/redis/6379.conf--700 That's ok -- modify , Turn on AOFappendonly yes--704 That's ok -- Appoint A0F File name appendfilename "appendonly.aof"--796 That's ok -- Whether to ignore the last instruction that may have problems aof-load-truncated yes/etc/init.d/redis_6379 restart    Restart the service 

4.2 Execute the process

Because of the need to record Redis Every command written by , therefore A0F No need to trigger ,AOF The execution process of is as follows ：
Order to append (append)： take Redis The write command of is appended to the buffer aof_ buf;
File is written to (write) Synchronize with files (sync)： According to different synchronization strategies, we will aof_buf The contents of the file are synchronized to the hard disk ;
File rewriting (rewrite)： Rewrite regularly AOF file , Achieve the purpose of compression

4.2.1 Order to append （append）

Redis First append the write command to the buffer , Instead of writing directly to a file , It is mainly to avoid writing commands to the hard disk directly every time , Cause the hard disk IO Become Redis The bottleneck of the load .
The format of the command append is Redis The protocol format of the command request , It's a plain text format , Good compatibility 、 High readability 、 Easy to handle 、 It is easy to operate and avoid secondary overhead . stay A0F In file , Except for the specified database select command ( Such as select0 For the selection 0 The database ) By Redis Added , The others are written commands sent by the client

4.2.2 File is written to (write) Synchronize with files (sync)

Redis Provides a variety of AOF Cache synchronization strategy , The policy involves the operating system write Functions and fsync function , The explanation is as follows ：
In order to improve the efficiency of file writing , In modern operating systems , When the user calls write Function to write data to a file , Operating systems usually store data in a memory buffer , When the buffer is filled or exceeds the specified time limit , To write the buffer data to the hard disk . This kind of operation improves the efficiency , But it also brings security issues : If the computer goes down , Data in the memory buffer will be lost ; So the system also provides fsync、fdatasync And so on , It can force the operating system to write the data in the buffer to the hard disk immediately , So as to ensure the security of data .
AOF There are three synchronization methods for the cache synchronization file policy , They are ：

vim /etc/redis/6379.conf

---729---
● appendfsync always:
explain ： Command write aof_ buf Call the system immediately after fsync The operation is synchronized to AOF file ,fsync After completion, the thread returns .
In this case , Every time you have a write command, you have to synchronize to AOF file , Hard disk IO Become a performance bottleneck ,Redis Only about a few hundred TPS write in ,
It's seriously reduced Redis Performance of ; Even with solid state drives (SSD), It can only process tens of thousands of commands per second , And it will greatly reduce SSD Life span of .

● appendfsync no:
explain ： Command write aof_ buf After calling the system write operation , incorrect AOF File do fsync Sync ;
Synchronization is the responsibility of the operating system , Usually the synchronization period is 30 second . In this case , The timing of file synchronization is uncontrollable ,
And there will be a lot of data piled up in the buffer , Data security cannot be guaranteed .

● appendfsynceverysec:
explain ： Command write aof_ buf After calling the system write operation ,write After completion, the thread returns ;
fsync The synchronization file operation is called once per second by a dedicated thread .
everysec It's a compromise between the two strategies , It's a balance between performance and data security ,
So it is Redis Default configuration , It's also our recommended configuration .

4.2.3 File rewriting (rewrite)

Over time ,Redis More and more write commands are executed by the server ,AOF The files will get bigger and bigger ： Too much AOF Files will not only affect the normal operation of the server , It can also cause data recovery to take too long .
File rewriting refers to periodic rewriting AOF file , Reduce AOF Volume of file . It should be noted that ,AOF Rewriting is to put Redis In process data is converted to write commands , Sync to the new AOF file ; Not to the old AOF File for any read 、 Write operation !
Another thing to note about file rewriting is ： about AOF In terms of persistence , File rewriting is highly recommended , But it's not necessary ; Even if there is no file rewriting , Data can also be persisted and stored in Redis Import at startup ： So in some implementations , Automatic file rewriting is turned off , And then through the timing task, at a certain time of the day .
The reason why file rewriting can be compressed AOF file , The reason lies in ：
Expired data is no longer written to the file ;
Invalid commands are no longer written to the file ： If some data is set repeatedly (set mykey test1, set mykey test2)、 Some data has been deleted (sadd myset vtest, del myset) etc. .
Multiple commands can be combined into one ： Such as sadd myset test1, sadd myset test2, sadd myset test3 Can be combined into sadd myset test1 test2 test3
From the above, we can see that , Because after rewriting AOF There are fewer orders to execute , File rewriting can reduce the space occupied by the file , It can also speed up recovery .

4.3 Trigger of file rewriting

Trigger of file rewriting , It is divided into Manual and automatic triggering
Manual trigger ： Call directly bgrewriteaof command , The execution of the order is related to bgsave Some similar : All are fork The subprocess does the specific work , And only in fork Time blocking .
Automatic triggering ： By setting auto-aof-rewrite-min-size Options and auto-aof-rewrite-percentage Option to automate BGREWRITEAOF. Only when auto-aof-rewrite-min-size and auto-aof-rewrite-percentage When both options are met , Will automatically trigger AOF rewrite , namely bgrewriteaof operation .

vim /etc/redis/6379.conf
----771----
auto-aof-rewrite-percentage 100
# At present AOF file size ( namely aof_current_size) Is the last time the log was rewritten AOF file size (aof_base_size) At twice , happen BGREWRITEAOF operation
auto-aof-rewrite-min-size 64mb
# At present A0F File execution BGREWRITEAOF The minimum value of the command , Avoid starting Reids Due to the small file size, frequent BGREWRITEAOF

4.4 Rewrite the process Note

About the file rewriting process , There are two points to note ：
（1） Rewriting by the parent process fork The subprocess goes on ;
（2） During rewrite Redis Write commands executed , Need to be added to the new AOF In file , So Redis Introduced aof_rewrite_buf cache

4.5 File rewriting process

（1）Redis The parent process first determines whether there is currently executing bgsave/bgrewriteaof Subprocesses , If there is one bgrewriteaof The command returns directly to , If there is bgsave Orders wait bgsave Execute after the execution is completed
（2） Parent process execution fork Action create subprocess , In this process, the parent process is blocked
（3.1） The parent process fork after ,bgrewriteaof Command return background append only file rewrite started Information doesn't block the parent process anymore
（3.2） because fork The operation uses copy on write technology , Child processes can only share fork Memory data during operation . Because the parent process is still responding to commands , therefore Redis Use AOF Rewrite buffer (aof_ rewrite_buf) Save this data , Prevent new AOF This data is lost during file generation . in other words ,bgrewriteaof perform period ,Redis The writing of The command is also appended to aof_ buf and aof_ rewirte_ buf Two buffers
（4） Sub process according to memory snapshot , Write to the new... According to the command merge rule AOF file
（5.1） The subprocess writes a new AOF After the document , Signal the parent process , Parent process update statistics , The details can be obtained through info persistence see
（5.2） The parent process put AOF The data in the rewrite buffer is written to the new AOF file , This guarantees new AOF The database state saved by the file is consistent with the current state of the server
（5.3） Use the new AOF File replace old file , complete AOF rewrite

4.6 Load on startup

When AOF On ,Redis Qi Priority will be given to loading AOF File to recover data ; Only when AOF closed , Will load RDB File recovery data .
When AOF Turn on , but AOF writing When piece does not exist , Even if RDB The file will not load if it exists .
Redis load AOF When you file , Would be right AOF Check the file , If the file is corrupted , Error will be printed in the log ,Redis Boot failure . But if it is AOF The end of the file is incomplete ( The end of the file is not complete when the machine goes down suddenly ), And aof-load- truncated Parameter on , A warning will be output in the log ,Redis Ignore AOF End of file , Successful launch .
aof-load-truncated The parameter is on by default

---

5、 ... and ： RDB and AOF Advantages and disadvantages

5.1 RDB Advantages and disadvantages of persistence

advantage ： RDB The files are compact , Small volume , Network transmission is fast , Suitable for full replication ; Recovery speed ratio AOF Much faster . Of course , And AOF comparison , RDB most One of the important advantages is that the impact on performance is relatively small .
shortcoming ： RDB The fatal disadvantage of the file is that the persistence mode of the data snapshot determines that the real-time persistence cannot be achieved , And today, data is more and more important , The massive loss of data is often unacceptable , therefore AOF Persistence becomes the mainstream . Besides ,RDB The file needs to be in a specific format , Compatibility is poor ( Like the old version Redis Not compatible with the new version of RDB file ). about RDB Persistence , On the one hand is bgsave It's going on fork In operation Redis The main process will block , On the other hand , Sub process writing data to the hard disk will also bring IO pressure .

5.2 AOF Advantages and disadvantages of persistence

And RDB Persistence corresponds to ,AOF The advantage of this is that it supports second level persistence 、 Compatibility is good. , The disadvantage is that the files are large 、 Slow recovery 、 Great impact on performance .
about AOF Persistence , The frequency of writing data to the hard disk is greatly increased (everysec Second level under strategy ),IO More stressful , It may even cause AOF Additional blocking problem .
AOF Document rewriting and RDB Of bgsave similar , There will be fork The blocking and the subprocess's I0 The pressure problem . relatively speaking , because AOF Write data to the hard disk more often , So right. Redis The performance of the main process will be affected more .

---

6、 ... and ：Redis Performance management

6.1 see redis Memory usage

info memory

6.2 Memory fragmentation rate

The value of memory allocated by the operating system used_memory_rss Divide Redis The memory value used used_memory calculated
Memory fragmentation is an inefficient allocation by the operating system / Caused by reclaiming physical memory ( Discontinuous physical memory allocation )

6.3 Track memory fragmentation rate

Track memory fragmentation rate for understanding Redis The resource performance of an instance is very important ：
The memory fragmentation rate is slightly higher than 1 It's reasonable , This value indicates that the memory fragmentation rate is relatively low
The memory fragmentation rate exceeds 1.5, explain Redis It consumes the physical memory 150 Number , among 50 Number is the memory fragmentation rate . Need to be in redis-cli Tools . On the input shutdown save command , And restart Redis The server .
The memory fragmentation rate is lower than 1 Of , explain Redis Memory allocation exceeds physical memory , The operating system is swapping memory . Need to increase available physical memory or reduce Redis Memory footprint .

6.4 Memory usage

redis The memory usage of the instance exceeds the maximum available memory , The operating system will start memory and memory swap Space exchange .
Ways to avoid memory swapping ：
● Select the installation for the cache data size Redis example
● Use as much as possible Hash Data structure storage
● Set up key The expiration time of

6.5 Internal recovery key

Ensure reasonable distribution redis Limited memory resources .
When the set maximum threshold is reached , You need to select one key Recycling strategy for , By default, the recycling policy is to prohibit deletion . Modify... In configuration file maxmemory-policy Property value ：

vim /etc/redis/6379.conf--598--maxmemory-policy noenviction  # Modify... In configuration file max-memory-policy Property value ●volatile-lru       : Use LRU The algorithm eliminates the data from the data set with the expiration time set ●volatile-ttl       : Select the data that is about to expire from the data set with expiration time ●volatile-random    : Randomly select data from the data set with expiration time set ●allkeys-lru        : Use LRU Algorithms eliminate data from all data sets ●allkeys-random     : Select any data from the data set ●noenviction        : Ban data obsolescence 

attribute	meaning
volatile-lru	Use LRU The algorithm eliminates the data from the data set with the expiration time set
volatile-ttl	Select the data that is about to expire from the data set with expiration time
volatile-random	Randomly select data from the data set with expiration time set
alkeys-lru	Use LRU Algorithms eliminate data from all data sets
allkeys-random	Select any data from the data set
noenviction	Ban data obsolescence

---

7、 ... and ： summary

Through the study of this chapter, we should know redis The meaning of high availability ： High availability refers to the time when the server can be accessed normally , The measure is how long a normal service can be provided . form ： Technologies for high availability include persistence 、 Master slave copy 、 Sentinels and groups
Focus on understanding persistence in high availability :RDB And AOF
（1） Persistence mode ： ①RDB： Periodic snapshots ②AOF： Near real-time persistence （ With everysec The way ）
（2）redis Enabled priority AOF > RDB And only if AOF When the function is off ,redis Will be used when restarting RDB The way to recover
（3）RDB and AOF Persistence pattern in ①RDB： from redis The main process （ periodic ）fork Derive the child process pair redis Data in memory is persisted , Generate to .rdb In file . ②AOF： According to the persistence strategy （alawys、no、everysec（ Default ））, First the redis The statements in are saved in the cache , Then synchronize from the buffer to .aof In file .
redis Your recovery strategy / advantage redis Similar to other common non relational databases , When data is stored in memory and stored in memory , When redis restart , Memory data loss , but redis adopt RDB or AOF The persistence function of can be in redis After restart , Priority read AOF file , be based on AOF File data recovery is used to “ Persistent save ” data .
author ： Sweet and sour · Small platoon Gu

Game programming , A game development favorite ~

If the picture is not displayed for a long time , Please use Chrome Kernel browser .