Mysql35 master slave replication

One . Master slave replication overview

1.1. How to improve database concurrency

Besides , General applications are for databases “ Read more and write less ”, In other words, the pressure on the database to read data is relatively large , One idea is to adopt the scheme of database cluster , do Master slave architecture 、 Conduct Read / write separation , This can also improve the concurrent processing ability of the database . But not all applications need to set the master-slave architecture of the database , After all, setting up the architecture itself has a cost .
If our goal is to improve the efficiency of highly concurrent database access , So the first consideration is how to Optimize SQL And index , It's simple and effective ; The second is to adopt Caching strategy , For example, use Redis Save hotspot data in memory database , Improve the efficiency of reading ; The last step is to adopt Master slave architecture , Read and write separation .

1.2. The role of master-slave replication

The master-slave synchronization design can not only improve the throughput of the database , And the following 3 All aspects of the role .
The first 1 A role ： Read / write separation .

The first 2 One function is data backup .
The first 3 One function is high availability .

Two . The principle of master-slave replication

Slave From Master Read binlog To synchronize data .

2.1. Principle analysis

Three threads :
In fact, the principle of master-slave synchronization is based on binlog For data synchronization . In the master-slave replication process , Will be based on 3 Threads To operate , A main library thread , Two slave threads .

Binary log dump thread （Binlog dump thread） It's a main library thread . When connecting from the library thread , The master library can send the binary log to the slave library , When the main library reads Events （Event） When , Will be in Binlog On Lock , After the read is complete , Release the lock .
Slave Library I/O Threads Will connect to the main library , Send request update to main library Binlog. At this point from the library I/O The thread can read the binary log of the main library and dump the data sent by the thread Binlog Update section , And copy it to the local relay log （Relay log）.
Slave Library SQL Threads The relay log from the library is read , And execute the events in the log , Keep the data in the slave library synchronized with the master library .

Copy three steps :
step 1： Master Log writes to binary log （ binlog ）.
step 2： Slave take Master Of binary log events Copy to its trunk log （ relay log ）;
step 3： Slave Redo events in relay log , Apply the changes to your own database . MySQL Replication is asynchronous and serialized , And restart from Access point Start copying .
The problem of replication :
The biggest problem with copying ： Time delay

2.2. The basic principle of reproduction

• Every Slave only one Master.
• Every Slave There can only be one unique server ID.
• Every Master There can be multiple Slave.

3、 ... and . One master and one slave architecture

a host Used to handle all Write requests , a Slave Be responsible for all Read request , The architecture is as follows ：

3.1. preparation

1. Get ready 2 platform CentOS virtual machine .
2. Each virtual machine needs to be installed MySQL ( It can be MySQL8.0 ).
   Can be in one CentOS Installed on MySQL, And then clone to copy 1 Table contains MySQL Virtual machine .
   Be careful ： The cloning method needs to modify the name of the newly cloned host ：① MAC Address ② hostname ③ IP Address ④ UUID .
   Besides , Virtual machines generated by cloning （ contain MySQL Server）, Then the cloned virtual machine MySQL Server Of UUID identical , Must be modified , Otherwise, an error will be reported in some scenes . such as ： show slave status\G , Report the following error ：

Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have
equal MySQL server UUIDs; these UUIDs must be different for replication to work.

modify MySQL Server Of UUID The way ：

vim /var/lib/mysql/auto.cnf
systemctl restart mysqld

3.2. Host configuration file

Suggest mysql Version consistent and running as a service in the background , All master-slave configuration items are configured in [mysqld] Under the node , And they're all small letters .
The specific parameter configuration is as follows ：

• Mandatory

#[ must ] The primary server is unique ID
server-id=1
#[ must ] Enable binary logging , Named path . such as ： My own local path /log/mysqlbin
log-bin=zyy-bin

• Optional

#[ Optional ] 0（ Default ） Read and write （ host ）,1 Indicates read-only （ Slave ）
read-only=0
# Set the retention time of log files , The unit is seconds 
binlog_expire_logs_seconds=6000
# Control the size of a single binary log . The maximum and default values for this parameter are 1GB
max_binlog_size=200M
#[ Optional ] Set up the database not to be copied 
binlog-ignore-db=test
#[ Optional ] Set up the database to be copied , Default all records . such as ：binlog-do-db=atguigu_master_slave
binlog-do-db= The name of the master database to be copied 
#[ Optional ] Set up binlog Format 
binlog_format=STATEMENT

3.2.1. binlog Format is set ：STATEMENT Pattern

Format 1： STATEMENT Pattern （ be based on SQL Copy of sentences (statement-based replication, SBR)）

binlog_format=STATEMENT

Each one will modify the data sql The statement will record binlog in . This is the default binlog Format .

• SBR The advantages of ：

-  Has a long history , Mature technology 
-  You don't need to record every line change , Less binlog Log volume , Smaller files 
- binlog All database changes are included in , According to this, we can audit the security of the database 
- binlog Can be used for real-time restore , Not just for copying 
-  The master-slave version can be different , The slave version can be higher than the master version 

• SBR The shortcomings of ：

-  Not all UPDATE Statements can be copied , Especially when there are uncertain operations 

• Statements using the following functions cannot also be copied ：LOAD_FILE()、UUID()、USER()、FOUND_ROWS()、SYSDATE() ( Unless enabled at startup --sysdate-is-now Options )

- INSERT ... SELECT  Generation ratio  RBR  More row level locks 
-  Replication requires a full table scan (WHERE  The index is not used in the statement ) Of  UPDATE  when , Need to be compared with  RBR  Request more row level locks 
-  For having  AUTO_INCREMENT  Field  InnoDB As far as the watch is concerned ,INSERT  Statements will block other  INSERT  sentence 
- For some complex statements , The resource consumption on the slave server will be more serious , and  RBR  In mode , It only affects the record of the change 
-  Execute complex statements if there is an error , Will consume more resources 
-  The data table must be almost consistent with the primary server , Otherwise, replication errors may occur 

3.2.2. binlog Format is set :ROW Pattern （ Line based replication (row-based replication, RBR)）

binlog_format=ROW

5.1.5 Version of MySQL Just started to support , Don't record every sql The context information of the statement , Only record which data has been modified , What's the change .

• RBR The advantages of ：

-  Any situation can be replicated , This is the most important for replication   Safe and reliable   Of .（ such as ： There will be no stored procedures under certain circumstances 、function、trigger The call and trigger of cannot be copied correctly ）
-  Most of the time , From the table on the server if there is a primary key , Replication will be much faster 
-  There are fewer row locks when copying the following statements ：INSERT ... SELECT、 contain  AUTO_INCREMENT  Field  INSERT、 With the no strings attached or many records not modified  UPDATE  or  DELETE  sentence 
-  perform  INSERT,UPDATE,DELETE  Fewer locks on statement 
-  From the server   Multithreading   To perform replication 

• RBR The shortcomings of ：

- binlog  Much more 
-  Complex rollback  binlog  It will contain a lot of data 
-  Execute... On the primary server  UPDATE  When the sentence is , All the changes will be recorded in  binlog  in , and  SBR  Only once , This can lead to frequent  binlog  The concurrent write problem of 
-  from  binlog  See what statements are copied in 

3.2.3. binlog Format is set :MIXED Pattern （ Hybrid mode replication (mixed-based replication, MBR)）

binlog_format=MIXED

from 5.1.8 Version start ,MySQL Provides Mixed Format , It's actually Statement And Row The combination of .
stay Mixed In mode , General statement modification and use statment Format preservation binlog. Such as some functions ,statement Unable to complete the master-slave replication operation , Then row Format preservation binlog.
MySQL According to each specific sql Statement to distinguish the log form to be recorded , That is to say Statement and Row Choose between .

3.3. Slave configuration file

All master-slave configuration items are required to be configured in my.cnf Of [mysqld] Under the field , And they're all small letters .

• Mandatory

#[ must ] From the server only ID
server-id=2

• Optional

#[ Optional ] Enable relay logging 
relay-log=mysql-relay

Restart the background mysql service , Make configuration effective .

 Be careful ： The master and slave computers close the firewall 
service iptables stop #CentOS 6
systemctl stop firewalld.service #CentOS 7

3.4. host ： Establish an account and authorize

# Host computer MySQL Execute the command authorizing master-slave replication in 
GRANT REPLICATION SLAVE ON *.* TO 'slave1'@' From the machine database IP' IDENTIFIED BY 'abc123';
#5.5,5.7

Be careful ： If you are using MySQL8, You need to establish an account in the following ways , And authorize slave：

CREATE USER 'slave1'@'%' IDENTIFIED BY '123456';
GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'%';
# This statement must execute . Otherwise, see below .
ALTER USER 'slave1'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
flush privileges;

If an execution error occurs , Execute first :

show variables like '%password%';

Just modify the password policy , There is an introduction before .
Inquire about Master The state of , And record File and Position Value .

show master status;

Record File and Position Value

 Be careful ： Do not operate the master server after this step MySQL, Prevent the state value of the primary server from changing .

3.5. Slave ： Configure the host to be replicated

step 1： The command to copy the host from the machine ：

CHANGE MASTER TO
MASTER_HOST=' The host IP Address ',
MASTER_USER=' Host username ',
MASTER_PASSWORD=' Password of host user name ',
MASTER_LOG_FILE='mysql-bin. Specific figures ',
MASTER_LOG_POS= Specific value ;

give an example ：

CHANGE MASTER TO MASTER_HOST='192.168.101.30',MASTER_USER='slave1',MASTER_PASSWORD='123456',MASTER_LOG_FILE='zyy-bin.000004',MASTER_LOG_POS=156;

step 2：

# start-up slave Sync 
START SLAVE

If you make a mistake ：
You can do the following , Delete the previous relay_log Information . Then execute again CHANGE MASTER TO … Sentence can be used .

mysql> reset slave; # Delete SLAVE Database relaylog Log files , And reactivate the new relaylog file 

next , View synchronization status ：

SHOW SLAVE STATUS\G;

The above two parameters are Yes, The master-slave configuration is successful ！
The following cases are explicitly , It's just not right . Possible reasons for the error are ：

1.  The Internet is not working 
2.  Wrong account password 
3.  A firewall 
4. mysql Configuration file problem 
5.  Syntax when connecting to the server 
6.  master server mysql jurisdiction 

3.6. test

Host new library 、 new table 、insert Record , Copy from the machine ：

CREATE DATABASE zyy_master_slave;
CREATE TABLE mytbl(id INT,NAME VARCHAR(16));
INSERT INTO mytbl VALUES(1, 'zhang3');

Host query ：

select * from mytbl;

Slave query ：

show database;

use zyy_master_slave;
show tables;

select * from mytbl;

3.7. Stop master-slave synchronization

• Stop master-slave synchronization command ：

stop slave;

• How to reconfigure master-slave
  If you stop copying from the server , Reuse requires reconfiguration of master-slave . Otherwise, the following error will be reported ：
  
  Reconfigure master-slave , Need to execute on the slave ：

stop slave;
reset master; # Delete Master All of the binglog file , And empty the log index file , Restart all new log files ( Use with caution )

Four . Synchronization data consistency problem

Requirements for master-slave synchronization ：

• The data of read library and write library are consistent ( Final agreement );
• Write data must be written to the write library ;
• To read data, you must go to the Reading Library ( not always );

4.1. Understand the master-slave delay problem

The content of master-slave synchronization is binary log , It's a file , It's going on Network transmission In the process of There is a master-slave delay （ such as 500ms）, In this way, the data that users read from the library may not be the latest data , That is, in master-slave synchronization Data inconsistency problem .

4.2. Cause of master-slave delay problem

When the network is normal , The time required to transfer logs from the master database to the slave database is very short , namely T2-T1 The value of is very small . namely , Under normal network conditions , The main source of the delay between the master and the standby is that the standby database has finished receiving binlog And the time difference between the execution of the transaction .
The most direct manifestation of active / standby delay is , Consume relay logs from the library （relay log） The speed of , It's better than the main library binlog It's slower . Cause ：

1、 The machine performance of the slave library is worse than that of the master library 
2、 High pressure from the reservoir 
3、 Execution of big business 

give an example 1： One time use delete Statement to delete too much data
Conclusion ： When deleting data later , To control the amount of data deleted per transaction , Delete multiple times .
give an example 2： One time use insert…select Insert too much data
give an example :3： The big table DDL
For example, check a card in the main library 500W Adding a field to your table takes 10 minute , Then the slave node will also consume 10 minute .

4.3. How to reduce master-slave delay

If you want to reduce the master-slave delay time , We can take the following measures ：

1. Reduce the probability of concurrency of multithreaded large transactions , Optimize business logic
2. Optimize SQL, Avoid slow SQL, Reduce batch operations , It is recommended to write scripts to update-sleep It's done in this way .
3. Improve the configuration of slave machines , Reduce the main database write binlog And reading from the library binlog Poor efficiency .
4. Use as far as possible Short link , In other words, the distance between the master database and the slave database server should be as short as possible , Increase port bandwidth , Reduce binlog transmission
   Network delay .
5. The real-time requirements of business read forced to go to the main database , Only do disaster recovery from the library , Backup .

4.4. How to solve the problem of consistency

If the data of the operation is stored in the same database , So when updating the data , You can write lock the record , In this way, data inconsistency will not occur when reading . But at this time, the function of slave library is Backup , It didn't work Read / write separation , Share the reading pressure of the main library The role of .

In the case of separation of reading and writing , Solve the problem of inconsistent data in master-slave synchronization , Is to solve the problem between master and slave Data replication The problem of , If according to data consistency From weak to strong To divide , There are the following 3 Two replication methods .

4.4.1. Method 1： Asynchronous replication

4.4.2. Semi-synchronous replication

4.4.3. Method 3： Group replication

Neither asynchronous replication nor semi synchronous replication can ultimately ensure data consistency , Semi synchronous replication determines whether to return to the client by judging the number of responses from the library , Although data consistency is improved compared with asynchronous replication , However, it is still unable to meet the scenarios with high requirements for data consistency , For example, in the financial sector .MGR It makes up for the shortcomings of these two replication modes .
Group replication technology , abbreviation MGR（MySQL Group Replication）. yes MySQL stay 5.7.17 A new data replication technology introduced in version , This replication technology is based on Paxos Protocol state machine replication .

• MGR How it works
  First, we form a replication group with multiple nodes , stay Perform reading and writing （RW） Business When , Need to pass the consistency protocol layer （Consensus layer ） Consent of , That is, the read-write transaction wants to commit , You have to go through the group “ Most people ”（ Corresponding Node node ） Consent of , Most mean that the number of agreed nodes needs to be greater than （N/2+1）, In this way, you can submit , Instead of the original sponsor has the final say. . And for the read-only （RO） Business It does not need to be approved by the group , direct COMMIT that will do .
  There are multiple nodes in a replication group , Each maintains its own copy of the data , Atomic messages and globally ordered messages are implemented in the consistency protocol layer , So as to ensure the consistency of data in the group .
  

MGR take MySQL Into the era of strong data consistency , It is an epoch-making innovation , One of the important reasons is MGR Is based on Paxos Agreed .Paxos The algorithm is based on 2013 The winner of the Turing prize in Leslie Lamport On 1990 Put forward in , The decision-making mechanism of this algorithm can be searched . in fact ,Paxos After the algorithm is proposed, it will be used as Distributed consistency algorithm Widely used , such as Apache Of ZooKeeper Is based on Paxos Realized .