Read the configuration, explain the principle and read the interview questions. I can only help you here...

When I was writing

As soon as I came up, I was the master and slave 、 colony 、 sentry , Who can stand it

When , Many friends can't wait to leave messages to see these models , Today we will start from the configuration file 、 Design principle 、 Let's talk about the real interview questions from three aspects  Redis  Master-slave replication of .

stay  Redis  On the basis of replication , Use and configure

Master slave copy

It's simple , Can make  Redis  The server （ Hereinafter referred to as  replica） Can accurately copy the master  Redis  The server （ Hereinafter referred to as  master） The content of . Every time when  replica  and  master  When the connection between is disconnected , replica  Will automatically reconnect to  master  On , And no matter during this period  master  What happened? , replica  Will try to be  master  An exact copy of .

Master slave copy , from  5.0.0  Version start ,Redis  Formal will  SLAVEOF  The order was changed to  REPLICAOF  Order and gradually abandon the original  SLAVEOF  command

Redis Use default asynchronous replication , Its characteristics are

Low latency

and

High performance

, It's the vast majority  Redis  The natural replication pattern of use cases . however ,replica  Will asynchronously confirm that it is from the master  master  The amount of data received in the cycle .

Master slave Topology

master  Used to write operations ,replicas  Used to read data , It is suitable for reading more and writing less . For scenarios with high write concurrency , Multiple slave nodes will cause the master node to send multiple write commands, which will consume the network bandwidth , It also aggravates  master  The load affects service stability .

replica  Other... Are acceptable  replica  The connection of . Except for many  replica  Can be connected to the same  master  outside , replica  It can also be like a laminated structure （cascading-like structure） Connect to other  replica . since  Redis 4.0  rise , be-all  sub-replica  Will be from  master  Receive exactly the same copy stream .

When  master  Multiple required  replica  when , In order to avoid being against  master  Performance interference , The tree master-slave structure can be used to reduce the pressure of the master node .

In order to make everyone have a clearer understanding of the concept , Let's take a look at the parameters of master-slave replication in the configuration file ：

REPLICATION

replicaof 

<masterip> <masterport>

By setting  master  Of  ip  and  port , Can make the current  Redis  The instance becomes another  Redis  A copy of the instance . stay Redis Startup time , It will automatically follow  master  Data synchronization .

• Redis  Replication is asynchronous , It can be modified by  master  Configuration of , stay  master  Not associated with the given number of  replica  When the connection , The host stops receiving writes ;

• If the replication link is lost for a relatively short time ,Redis replica  It can be done with  master  Perform partial resynchronization , Can use reasonable  backlog  Value to configure (
  See below
  );

• Replication is automatic , No user intervention is required . After the network partition ,replica  Will automatically attempt to reconnect to  master  And with  master  To resynchronize ;

masterauth 

<master-password>

When  master  When password protection is set ,replica  Service connection  master  Password

replica-serve-stale-data yes

When  replica  And  master  Loss of connection or master-slave replication in progress ,replica  There can be two different settings ：

• replica-serve-stale-data：yes（ The default value is ）, be  replica  Will still respond to client requests , There may be expired data , Or if this is the first synchronization , Then the dataset may be empty .

• replica-serve-stale-data：no , replica  All requested commands will be （ But does not contain  INFO, replicaOF, AUTH, PING, SHUTDOWN, REPLCONF, ROLE, CONFIG, SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PUBLISH, PUBSUB, COMMAND, POST, HOST: and LATENCY） return  
  SYNC with master in progress
    Error of .

replica-read-only

Can be  replica  Configure to read only ,yes  Represents a read-only state , All write commands will be rejected ;no  Indicates that you can write .  from  Redis 2.6  after , replica  Read only mode is supported and enabled by default . You can use... At run time  

CONFIG SET

  To turn on or off at any time .

Yes  replica  Writing may help store some

temporary

data （ Because writing  replica  The data is related to  master  It can be easily deleted after resynchronization ）, Calculating a slow set or sort set operation and storing it in a local key is a use case for multiple observed writable copies . But if the client writes data to it due to a configuration error , It may also cause problems .

stay

Cascade structure

Even if  replica B  Nodes are writable ,Sub-replica C  I won't see  B  Writing , But will have and  master A  The same dataset .

Set to  yes  This does not mean that the client is clustered to  replica  When connecting to the cluster for the portal , It can't be  set  operation , And  set  The data of the operation will not be placed in  replica  Trough , Will be put in some  master  Trough .

Be careful ： read-only  replica  It is not designed to expose untrusted clients on the Internet , It is just a protective layer against instance misuse . By default , Read only copies still export all administrative commands , Such as CONFIG、DEBUG  etc. . Up to a point , have access to

rename-command

To hide all management / Dangerous orders , This improves the performance of read-only copies

Security

.

repl-diskless-sync

Replication synchronization policy ： disk （disk） Or socket （socket）, The default is  no  Use  disk .

new  replicas  And reconnected  replicas  If the replication process cannot continue because a difference is received , You need to execute “ Full synchronization ”.RDB  File from the  master  Transferred to the  replicas, Transmission can be carried out in two different ways ：

• Disk-backed：Redis master  Node creates a new process and will  RDB  File is written to
  disk
  , The file is then incrementally transferred to... Through the parent process  replicas  node ;

• Diskless：Redis master  Node creates a new process and directly  RDB  The file is written to  replicas  Of  sockets  in , Do not write to disk .

• When doing  disk-backed  Replication time , RDB  File generation finished , Multiple  replicas  adopt
  line up
  Synchronization  RDB  file .

• When doing  diskless  Replication time ,master  The node will wait for some time （ The lower one repl-diskless-sync-delay  To configure ） Retransmission to expect multiple  replicas  Connect in , such  master  The node can
  Simultaneous synchronization
  To more than one  replicas  node . If the waiting time is exceeded , You need to line up , Wait for the current  replica  After processing, proceed to the next  replica  To deal with .

Poor hard disk performance , When the network performance is good  diskless  The effect is better.

Warning ： Diskless replication is currently in the experimental stage

repl-diskless-sync-delay

When you enable  diskless  After copying , You can set... With this option  master  The time that a node waits before creating a child process , That is, delay the start of data transmission , The purpose can be in the first  replica  When ready , Waiting for more  replica  be ready . The unit is in seconds , The default is

5 second

.

repl-ping-replica-period

Replica  send out  PING  To  master  The interval of , The default value is  10  second .

repl-timeout

The default value is 60 second , This option is used to set the following conditions  timeout  Judge ：

• from  replica  From the point of view of nodes  SYNC  In process  I/O  transmission  ——  Have not received  master SYNC  Transmission of  rdb snapshot  data ;

• from  replica  From the point of view of nodes  master  Of  timeout（ Such as  data,pings）—— replica  Have not received master Send packets or ping;

• from  master  From a node point of view  replica  Of  timeout（ Such as  REPLCONF ACK pings）—— master  Have not received  REPLCONF ACK  The confirmation of ; It should be noted that , This option must be greater than  repl-ping-replica-period, Otherwise, in the  master  and  replica  It often happens when there is low traffic between  timeout.

repl-disable-tcp-nodelay

master  and  replicas  Whether the connection of the node is turned off  TCP_NODELAY  Options .

• If you choose “yes”,Redis  Will use less  TCP  Packets and less bandwidth  replicas  send data . But this will increase the data in  replicas  End display delay , For... Using the default configuration  Linux  kernel , Delay reachability 40 millisecond .

• If you choose “no”, Then the data appears in  replicas  The end delay will be reduced , But replication will use more bandwidth .

What this actually affects is  TCP  Layer options , It's going to use  setsockopt  Set up , The default is  no, Express  TCP  Layers are disabled  Nagle  Algorithm , Send the data as soon as possible ,  Set to  yes  Express  TCP  Layer enable  Nagle  Algorithm , Data accumulated to a certain extent , Or after a certain time  TCP  Layer will emit .

By default , We will optimize for low latency , But when the flow is very high , Or when  master  and  replicas  Distance multiple  hops  when , Change this option to “yes” It might be better .

repl-backlog-size

Set the number of copies  backlog  Buffer size , Default  1mb.backlog  It's a buffer , When  replica  When disconnected for a while , It will accumulate  replica  data , So when  replica  When you want to reconnect again , In general, you don't need

Full amount of synchronization

, Just do it

Partial synchronization

that will do , Pass only  replica  Some data lost when disconnected .

Bigger  backlog  Buffer size , signify  replicas  After disconnection and reconnection , The longer it can still continue （ Support disconnection for longer time ）.

backlog  Buffer only in at least one  replica  When the node is connected  master  Nodes need to be created .

repl-backlog-ttl

When  replicas  After the node is disconnected ,master  The node will be released after a period of time  backlog  buffer .  This option is set when

the last one

 replica  After breaking the link ,master  How many seconds to wait before releasing the buffer . Default 3600  second ,0 Means never release .

replicas  The node will never release this buffer , Because it is possible to connect to  master  node ,  Then try  “ The incremental synchronization ”.

replica-priority

replica-priority  yes  Redis  adopt  INFO  The integer published by the interface , The default value is  100.  When  master  After the node fails to work properly  Redis Sentinel  This value determines which  replica  The node is promoted to  master  node . The numerical

The smaller it is

Express

Higher priority

To improve . If there are three  replica  Its node  priority  Values, respectively  10,100,25, Sentinel  Will choose  priority  by  10  The node is promoted . The value of  0  Express  replica  Nodes always

You can't

Promoted to  master  node .

min-replicas-to-write

min-replicas-max-lag

// Indicates that at least 3 A delay <=10 A copy of seconds exists
min-replicas-to-write 3 // In the following  N
min-replicas-max-lag 10 // In the following  M


from  Redis 2.8  Start , If connected  replica  The delay is less than or equal to M The number of seconds is less than N individual （N individual  replica  Need to be in “online” state ）, be  master  May stop accepting writes and reply  error. because  Redis  Use asynchronous replication , Therefore, there is no guarantee that  replica  Whether the given write command is actually received , So there will always be a data loss window .

The principle is as follows ：

• replica  Every second  ping master, Confirm the number of replication streams processed ;

• master  Will remember the last time from each  replica  All received  ping  Time for , Delay is based on  master  from  replica  Last time received  ping  Calculated ;

• The user can configure the minimum delay that does not exceed the maximum number of seconds  replica  Count ;

This option does not guarantee  N  Copies will accept writes , But when not enough copies are available , Limit the exposure window for lost writes to a specified number of seconds .

N  The default value is  0,M  The default value is 10. Any one is set to  0  Indicates that this function is not enabled .

replica-announce-ip 5.5.5.5

replica-announce-port 1234

Redis master  You can list the connected... In different ways  replicas  The address and port of the node .  Such as  Redis Sentinel  I'll use it later  “INFO replication”  Command to get  replica  Instance information ,master  Of “ROLE“  The command also provides this information .

This information is generally through  replica  The node obtains and reports the information in the following ways ：

• IP： Connect to by automatic identification  Socket  Automatic acquisition of information

• Port： Generally speaking, this value is  replicas  The listening port used by the node to accept the connection of the client

however , If port forwarding is enabled or  NAT, You may need another address and port to connect to  replicas  node .  In this case , You need to set these two options , such  replicas  The values set by these two options will override the values obtained by the default behavior , And then report it to  master  node .  According to the actual situation , You can only set one of these options , Instead of setting both options .

This concludes the introduction of configuration , Next, let's string together the concepts mentioned above , Talk about the principle of master-slave replication .

principle

The operation of the system depends on three main mechanisms

• When one  master  Instance and a  replica  When the instance connection is normal , master  A series of command streams will be sent to keep  replica  Update , In order to copy the changes of its own dataset to  replica , Include client writes 、key  Expired or expelled, etc .

• When  master  and  replica  After the connection between , Because of the network problem 、 Or master-slave
  consciousness
  Connection to timed out , replica  Reconnect on  master  And will try to do some resynchronization . This means that it tries to get only the command stream that was lost during the disconnect .

• When partial resynchronization is not possible , replica  Full resynchronization will be requested . This involves a more complex process , for example  master  You need to create a snapshot of all the data , Send it to  replica , After that, when the data set changes, the command stream will be continuously sent to  replica .

Redis  How replication works

every last  

Redis master

  There is one.  

replication ID

 ： This is a big

Pseudo random string

, Marked a given dataset . Every  master  Also holds an offset ,master  Send the self generated copy stream to  replica  when , How many bytes of data to send , How much the offset of itself will increase , The purpose is to modify your dataset when there are new operations , It can be updated with  replica  The state of .

Copy offsets even without one  replica  Connect to  master  when , It will also increase , So basically every given pair  

Replication ID, offset

  Will be marked with one  master  The exact version of the dataset .

When  replica  Connect to  master  when , It USES  PSYNC  Command to send the old  master replication ID  And the offset it has processed so far . In this way , master  Only... Can be sent  replica  The required incremental portion . But if  master  There is not enough... In the buffer of  backlog  perhaps  replica  Refer to the  master  Unknown historical records （replication ID）, Then a full resynchronization will be carried out instead ： under these circumstances , replica  You will get a complete copy of the dataset , Start from scratch .

Speaking of this , So what is full synchronization , What is incremental synchronization ？

Full amount of synchronization

• replica  Connect  master, send out  PSYNC  command ;

• master  perform  bgsave  Start a background save process , In order to produce a  RDB  file . At the same time, it starts buffering all new write commands received from the client .

• When the background save is complete , master  Transfer the dataset file to all  replica, And continue to record the write commands executed during the send ;

• replica  received  RDB  After the document , Discard all the old data , Then load the new file into memory ;

• replica  After loading , notice  master  Send all buffered commands to  replica, This process is completed in the form of instruction flow and is similar to  Redis  The format of the agreement itself is the same ;

• replica  Start receiving command requests , And execution from  master  Write command to buffer .

Be careful ：SYNC  It's an old agreement , In the new  Redis  Is no longer used in , But still backward compatible . Because it does not allow partial resynchronization , So now  PSYNC  Be used in place of  SYNC.

Under normal circumstances , A full resynchronization requires the creation of a... On disk  RDB  file , Then load it from disk into memory , then  replica  This is used for data synchronization . If disk performance is low , This is right  master  It's a very stressful operation .Redis 2.8.18  Is the first version to support diskless replication . In this setting , Subprocesses send directly  RDB  Document to  replica  Of  sockets  in , No need to use disk as intermediate storage medium .

The incremental synchronization

master  Send the command to all  replica  At the same time , Commands are also written to  backlog  In the buffer .

When  replica  And  master  After disconnecting and reconnecting , At this point, judge  replica  The offset of is the same as  master  Does the difference set of the offset exceed  backlog  Size ,

• If not, give  replica  send out  CONTINUE, wait for  master  take  backlog  The data in is sent to  replica;

• If it exceeds, return  FULLRESYNC runid offset,replica  take  runid  Save up , And conduct full synchronization ;

Finally, let's talk about some interview questions often mentioned in the interview process .

Interview questions

In the master-slave replication process , close  master  What problems will be caused by the persistence of ？

The data will come from  master  And all  replica  The deleted . Let's use a case to illustrate ：

• We set up nodes  A  by  master  And turn off its persistence settings , Set nodes  B  and  C  by  replica;

• When  master  Crash Time , Because the script of automatic restart is configured in the system , here  master  It will restart automatically . But because persistence is turned off ,master  After restart, its data set is empty ;

• here , If  replica  from  master  Data synchronization in , It will lead to  replica  The data in will also become an empty set .

therefore , We are using  Redis  When copying functions ,

Strongly recommended

stay  master  and  replica  Enable persistence... In . If persistence is not enabled because of the delay caused by very slow disk performance ,

The node should be configured to avoid automatic restart after reset

.

Redis  How to deal with replication  key  The expiration of

Redis  The expiration mechanism can limit  key  Survival time , The mechanism depends on  Redis  The ability to calculate time . however , Even using  Lua  The script will these  key  Become expired  key,Redis replicas  Can also copy these correctly  key.

In order to do that ,Redis  Cannot rely on master-slave use

Synchronous clock

, Because this is an unsolvable problem and will lead to  race condition  Inconsistent with the data set , therefore  Redis  Use three main techniques to make expired  key  Replication works correctly ：

• replica  Will not let  key  Be overdue , But wait  master  Give Way  key  Be overdue . When one  master  To make a  key  expire （ Or because of  LRU  The algorithm expels it ） when , It will synthesize a  DEL  Command and transfer to all  replica;

• Due to the main drive ,master  Unable to provide in time  DEL  command , So sometimes  replica  There may still be logically expired... In your memory  key. In order to deal with this problem ,replica  Use its logical clock to report without violating data consistency , Of the read operation  key  non-existent . by this means ,replica  Avoid reporting logic that is out of date  key  There is still . in application , Use  replica  Program extension  HTML  Fragment cache , It will avoid returning data items that are already earlier than expected .

• stay Lua During script execution , Do not execute any  key  Expiration action . When one Lua Script runtime , conceptually ,master  The time in is frozen , So when the script runs , A given key either exists or does not exist . This prevents  key  Expire in the middle of the script , Make sure to send the same script to  replica , This produces the same effect in both data sets .

Once a  replica  Promoted to a  master , It will begin to expire independently  key, Without any old  master  With the help of the .

o  Q  Will continue to update  java  Practical articles , If you have a different opinion or a better one  idea, Welcome to contact ah  Q.

【 o  Q  Say code 】, Notable official account

The style of the article is changeable , The pictures are easy to understand , The story is vivid and interesting , Let's talk about technology ！