MySQL master-slave synchronization principle

Master slave copy , It is used to build a database environment exactly the same as the main database , Called from database ; The main database is generally a quasi real-time business database .

principle ：

1. The database has a bin-log Binary , It records everything sql sentence .

2. Our goal is to make the main database bin-log Of documents sql Copy the sentence .

3. Let it in from the data of relay-log Redo these in the log file again sql Sentence can be used .

4. Three threads are needed to operate ：
   4.1 binlog Output thread : Whenever there is a slave connection to the master , The main library will create a thread and send binlog Content to from library . From the library , When replication begins , From the library, two threads will be created for processing ：
   4.2 Slave Library I/O Threads : When START SLAVE Statement is executed from the library , Create a... From the library I/O Threads , This thread connects to the main database and requests the main database to send binlog The update records in it are recorded on the slave database . Slave Library I/O The thread reads the main library binlog The output thread sends the updates and copies them to the local file , These include relay log file .
   4.3 From library SQL Threads : Create a... From the library SQL Threads , This thread reads from the library I/O The thread writes relay log Update event and execute .
   You can know , For each master-slave replication connection , There are three threads . The master database with multiple slave databases creates one for each slave database connected to the master database binlog Output thread , Each slave has its own I/O Threads and SQL Threads .

purpose ：

1、 Implement disaster recovery , Fail over

2、 Read / write separation

3、 Backup 、 Avoid data loss

Conditions ：

1、 Main library opens binlog journal

2、 Master slave server server-id Different

3、 The slave server can connect to the master database

The way ：

1、 Synchronous replication

Synchronous replication , intend master The change of , Must wait slave-1,slave-2,…,slave-n Don't go back until you've finished .

such , Obviously not desirable , Neither MYSQL Default settings for replication . such as , stay WEB On the front page , The user added records , It takes a long time to wait .

2、 Asynchronous replication

Like AJAX Request the same .master You only need to complete your own database operation . as for slaves Binary log received or not , Is the operation complete , Don't care. .MYSQL Default Settings .

3、 Semi-synchronous replication

master Ensure that only slaves One of the operations was successful , Just go back to , other slave No matter .

This function , By google by MYSQL Introduced .

Principle steps ：
Step one ： Main library db Update event for (update、insert、delete) Written to binlog

Step two ： Initiate connection from library , Connect to the main library

Step three ： At this time, the main library creates a binlog dump thread Threads , hold binlog Content sent from library

Step four ： After starting from the library , Create a I/O Threads , Read from the main library binlog Content and write to relay log.

Step five ： And create a SQL Threads , from relay log Read the contents inside , from Exec_Master_Log_Pos The location starts executing the read update event , Write the update to slave Of db.

About MySQL Some problems of master-slave synchronization

• master Write operations for ,slaves Passively do the same thing , Keep data consistent , that slave Whether you can write actively ？

hypothesis slave Can actively write operations ,slave And we can't inform master, This leads to master and slave The data are inconsistent . therefore slave There should be no writing , At least slave Database related to replication cannot be written on . actually , The concept of separation of reading and writing has been revealed here .

• Master slave replication , There can be N individual slave, But these slave You can't write , What do you want them to do ？

Data backup can be realized .

Similar to highly available features , once master Hang up , It can make slave Top up , meanwhile slave Upgrade to master.

Disaster tolerance in other places , such as master In Beijing, , The earthquake hung up , So in Shanghai slave You can go on .

Mainly used to realize scale out, Share the load , The task of reading can be spread to slaves On

• There are... In master-slave replication master,slave1,slave2,… Wait, so much MYSQL database , That's like a JAVA WEB Which database should the application connect to ?

We can do this in an application ,insert/delete/update These operations to update the database , use connection(for master) To operate ,select use connection(for slaves) To operate . Then our application has to complete how to start from slaves Choose one to perform select, For example, a simple round robin algorithm .

In this case , It's like the application is done SQL Statement routing , And with the MYSQL The master-slave replication architecture of is very relevant , once master Hang up , some slave Hang up , Then the application will be modified

• Can you make the app and MYSQL There's not much to do with the master-slave replication architecture of ？

Find a component ,application program Just deal with it , Use it to do MYSQL Agent for , Realization SQL Statement routing .

mysql proxy Not responsible , How to start from many slaves Pick one ？ You can give it to another component ( such as haproxy) To complete .

That's what's called MYSQL READ WRITE SPLITE,MYSQL Separation of reading and writing .

• If mysql proxy , direct , master What if some of them hang up ？

The president usually gets a vice president , provide against any misfortune . alike , You can give these critical nodes a backup .

• When master For each event generated by the binary log of , All need to be sent to slave, If we had N individual slave, That's hair N Time , Or just once ？

If only once , Issued to slave-1, that slave-2,slave-3,… What do they do ？

display however , It should be sent out N Time . actually , stay MYSQL master Inside , maintain N Threads , Each thread is responsible for sending binary log files to the corresponding slave.master Be responsible for the writing operation , The maintenance of return N Threads , The burden will be heavy . can In this way ,slave-1 yes master From the ,slave-1 again slave-2,slave-3,… The Lord of , meanwhile slave-1 No longer responsible for select. slave-1 take master The burden of replication on threads , Transfer to yourself . This is the concept of multi-level replication .

• When one select Sent to mysql proxy, Maybe this time by slave-2 Respond to , Next time by slave-3 Respond to , In this case , You can't take advantage of the query cache .

You should find a shared cache , such as memcache To solve . take slave-2,slave-3,… The results of these queries are cached to mamcache in .

• With the increasing application , There are a lot of reading operations , We can expand slave, But if master I can't satisfy the writing operation , What shall I do? ？

scale on ? Better servers ？ No best , Only the better , It's too expensive ...

scale out ? The master-slave replication architecture can no longer meet the requirements of .

• From the database read delay problem understand ？ How to solve ？

reason ： Main library TPS High concurrency ,DDL More than slave One sql The range of threads , It may also be caused by large queries , And network latency .（ talk about MySQL The principle of database master-slave synchronization delay , from mysql From the principle of replication ,mysql Master slave replication is a single threaded operation , The main library is for all DDL and DML produce binlog,binlog It's written in sequence , So it's very efficient ;slave Of Slave_IO_Running The thread will fetch logs from the main library , It will be more efficient ,slave Of Slave_SQL_Running Thread will be the main library of DDL and DML Operations are all in slave The implementation of .DML and DDL Of IO The operation is random , It's not sequential , Therefore, the cost will be very high , It could be slave Other queries on generate lock contention , because Slave_SQL_Running It's also single-threaded , So a DDL Card owner , You need to perform 10 minute , So all of the following DDL Will wait for this DDL It's not until it's done , This leads to delays . A friend will ask ：“ The same one on the main library DDL It also needs to be carried out 10 branch , Why? slave It's going to delay ？”, The answer is master Can the concurrent ,Slave_SQL_Running Threads can't .）

** Solution 1 ：** The simplest reduction slave The solution of synchronization delay is to optimize the architecture , Try to let the main warehouse DDL Fast execution . And the main database is to write , High data security , such as sync_binlog=1,innodb_flush_log_at_trx_commit = 1 Settings like that , and slave It doesn't need such high data security , It's all right to say sync_binlog Set to 0 Or shut down binlog,innodb_flushlog Can also be set to 0 To improve the sql Efficiency of execution . The other is to use a better hardware device than the main library as slave.

Solution 2 ： Put data into cache , After updating the database , In the case that it is expected to be used immediately , Proactive cache refresh .

Solution three ： For data that is important and must be real-time , For example, the user just changed his password （ Password write Master）, Then log in with your new password （ from Slaves Read password ）, It will cause password inconsistency , Cause the user to login in a short time . So in this case It is best to read real-time data from Master Direct reading , avoid Slaves Data lag occurs .

• How to do after the master server hangs up ？

Suppose there is an emergency ,master Downtime , Now the need is to 192.168.1.102 Upgrade the main library , The other is slave Library

step ：

1. Ensure that all relay log All updated , Execute on each slave Library stop slave io_thread; show processlist; Until we see Has read all relay log, The update from the library is completed

2. Log in to all slave libraries , see master.info file , Contrast selection pos Largest as new master , Here we choose 192.168.1.102 For the new main library

3. land 192.168.1.102, perform stop slave; And enter the database directory , Delete master.info and relay-log.info file , To configure my.cnf file , Turn on log-bin, If there is

log-slaves-updates and read-only Comment out , perform reset master

4. Create a user for synchronization and authorize slave, The same as the fifth step

5. Log in to another slave Library , perform stop slave Stop syncing

6. Connect to the new main library according to the seventh step

7. perform start slave;

8. Modify the new master data , test slave Whether to synchronize updates

• Implementation method of read-write separation ：

In order to reduce the pressure of database , Generally, the read-write separation of the database , The first implementation method is through analysis sql The sentence is insert/select/update/delete Which one of these , So as to select the master-slave , The second is to determine the master-slave by intercepting the method name , Such as ：save*()、insert*() The formal method uses master library ,select() The use of the beginning slave library .

Although most of them realize the separation of reading and writing directly from the program , But for distributed deployment and horizontal and vertical segmentation , Some agent software similar to middleware is quite practical , Such as MySQL Proxy Compare .mysql proxy There is no configuration file , lua Script is all about it , Of course lua It's quite convenient .

Related configuration ：

innodb_flush_log_at_trx_commit and sync_binlog yes MySQL Two configuration parameters . Their configuration is for MySQL The performance of （ Generally, in order to ensure that data is not lost , Will be set to double 1, In this case, the performance of the database is also the lowest ）.

• 1、innodb_flush_log_at_trx_commit

innodb_flush_log_at_trx_commit： yes InnoDB Engine specific ,ib_logfile How to refresh （ ib_logfile： The record is redo log and undo log Information about ）

Value :0/1/2

innodb_flush_log_at_trx_commit=0, Every second log buffer Brush into the file system (os buffer) Go to , And call the file system's “flush” Operation to flush the cache to disk . That is to say, the logs a second ago are stored in the log buffer , That is, in memory , If the machine goes down , May be lost 1 Seconds of transaction data .

innodb_flush_log_at_trx_commit=1, Indicates that every time a transaction is committed , All the log buffer Brush into the file system (os buffer) Go to , And call the file system's “flush” Operation to flush the cache to disk . In this case , Database pair IO The requirements are very high , If the underlying hardware provides IOPS Relatively poor , that MySQL Database concurrency will soon be due to hardware IO It's a problem that can't be raised .

innodb_flush_log_at_trx_commit=2, It means that every time a transaction is committed, it will put log buffer Brush it into the file system , But it doesn't immediately flash to disk . If it's just MySQL The database is down , Because there's no problem with the file system , Then the corresponding transaction data is not lost . Only when the host operating system where the database is located is damaged or suddenly powered down , The transaction data of the database may be lost 1 Seconds and so on . Such benefits , Reduce the probability of transaction data loss , And for the underlying hardware IO It's not that demanding (log buffer Write to the file system , It's usually just from log buffer Memory transfer of the file system in memory cache , To the underlying IO There is no pressure ).

• 2、sync_binlog

sync_binlog： yes MySQL Binary log （binary log） Frequency of synchronization to disk .

Value ：0-N

sync_binlog=0, When the transaction is committed ,MySQL Do not do fsync Such as disk synchronization instruction refresh binlog_cache Information in to disk , And let Filesystem Make your own decision when to synchronize , perhaps cache Sync to disk when full . This one has the best performance .

sync_binlog=1, When you go on 1 After transaction commit ,MySQL There will be one fsync Disk synchronization instructions like this will binlog_cache Data in is forced to write to disk .

sync_binlog=n, When you go on n After transaction commit ,MySQL There will be one fsync Disk synchronization instructions like this will binlog_cache Data in is forced to write to disk .

notes ：

Most of the time , There is no strict requirement for data consistency , So it's not going to take sync_binlog configure 1. In pursuit of high concurrency , Lifting performance , It can be set to 100 Or directly 0.

And he innodb_flush_log_at_trx_commit equally , For applications like payment services , It's still recommended sync_binlog = 1.

JVM Causes of memory leaks and memory overflows
JVM Interpretation and use of common monitoring tools
Redis Frequently asked questions （ One ）
ClickHouse And MaterializeMySQL engine （ Ten ）
Implementation and difference of three distributed locks
Understanding and use of thread pool

Out of order ！ Out of order ！

Recent interview BAT, Organize an interview document , covers Java The core technology 、JVM、Java Concurrent 、SSM、 Microservices 、 database 、 Data structure, etc . Want to get ？ If you want to improve yourself , And want to make progress with excellent people , Interested friends , You can scan the code below the official account. . The information is lying in the official account. ...

• Collect if you like
• Agree and praise
• Support is concerned about
• If in doubt, comment

Four in one , Yours offer Also four even

————————————————————————————————

The author of this article ：Java Technical debt
Link to the original text ：https://www.cuizb.top/myblog/article/1644849971
Copyright notice ： All articles in this blog except special statement , All adopt CC BY 3.0 CN License by agreement . For reprint, please sign the author and indicate the source of the article .