Deep parsing MySQL binlog

1. summary

binlog yes Mysql sever A binary log maintained by the layer , And innodb In the engine redo/undo log It's a completely different log ; It is mainly used to record all database table structure changes 、 And binary files for data modification , It doesn't record SELECT SHOW Wait for the operation ,Binlog With " Business " Is saved on disk , It also includes the elapsed time of statement execution ; It is mainly used in two scenarios ：
Master slave copy
Data recovery
Binlog The default file name of is “ Host name _binlog- Serial number ” Format , You can also specify a name in the configuration file . There are three kinds of file recording modes ：

ROW： The log records the modification of each line of data , And then again slave Modify the same data on both sides . Complete data synchronization 、 Data recovery . But there will be a lot of logs （ Reliable but resource consuming ）. Such as ： Mass data update 、 In especial alter table It's going to make the logs soar .

STATMENT： Every piece of modified data sql Will be recorded as master Of binlog in ,slave When copying sql The process resolves to the original master Same as that performed at the end sql Re execution . abbreviation sql Copy . Less logs 、 Less IO、 Improved storage and recovery , In some cases , It will lead to inconsistency between master and slave data . Such as ：now() last_insert_id().

MIXED： A mixture of the above two modes , In general use STATMENT Mode save binlog, about STATMENT Operations that cannot be copied use ROW Mode save binlog,MYSQL It will be executed according to sql Statement write mode .

2.Binlog File structure

Mysql Of binlog The file records all kinds of modification operations to the database , The data structure used to represent the modification operation is Log event. Different modification operations correspond to different log event. More commonly used log event Yes ：Query event 、Row event、 Xid event etc. . binlog The content of the document is all kinds of log event Set .

3.binlog Write mechanism of

Triggered by recording mode and operation event Event generation log event ( Event triggered execution mechanism ), Will be generated in the process of transaction execution log event Write buffer , Each transaction thread has a buffer .log event Save in a bin_cache_mngr In the data structure of , There are two buffers in the structure , One is stmt_cache, Used to store information that does not support transactions ; The other is trx_cache, Used to store information supporting transactions . The transaction commit phase will produce log event Write to external binlog In file , Different transactions connect in a serial manner log event Write to binlog In file , So a transaction contains log event The information is kept in binlog The file is continuous , No other transactions will be inserted in the middle log event.

4 The command operation - Restore data

stay /etc New under the directory my.cnf And add the following ：

*  /Users/zhaoshuai11 cd /etc
*  /etc cat my.cnf
[mysqld]
# log_bin
log-bin = mysql-bin # Turn on binlog
binlog-format = ROW # choice row Pattern 
server_id = 1 # To configure mysql replication Need to define , Unable to join canal Of slaveId repeat 

show variables like '%log_bin%';

obtain binlog File list ：

show binary logs;

Conditions for generating new log files ： Whenever we stop or restart the server , The server will log the log file to the next log file ,MySQL A new log file will be generated upon restart , File number increment . If the log file exceeds max_binlog_size（ The default value is 1G） The upper limit of the system variable configuration , A new log file will also be generated （ What we need to pay attention to here is , If you're using big business , Binary logs will also exceed max_binlog_size, No new log files will be generated , All transactions are written to a binary log , This situation is mainly to ensure the integrity of the transaction ） When the log is refreshed , Create a new log file

flush logs;

Execute the following statement :

create database if not exists test_binlog;

use test_binlog;


DROP TABLE IF EXISTS `Websites`;
CREATE TABLE `Websites` (
  `id` int(11) NOT NULL,
  `name` char(20) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
) DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `access_log`;
CREATE TABLE `access_log` (
  `aid` int(11) NOT NULL,
  `site_id` int(11) NOT NULL,
  `count` int(11) NOT NULL
);

INSERT INTO `Websites` VALUES ('1', 'Google');
INSERT INTO `Websites` VALUES('2', 'TaoBao');
INSERT INTO `Websites` VALUES('3', 'CaiNiao');

INSERT INTO `access_log` VALUES ('1', '1', '45');
INSERT INTO `access_log` VALUES('2', '3', '100');
INSERT INTO `access_log` VALUES('3', '1', '230');
INSERT INTO `access_log` VALUES('4', '2', '10');

show binary logs You can see the output as follows , This indicates that the operation we just performed has been written to mysql-bin.000007 in

We can use the following command to view what is currently being written binlog file ：

show master status\G

Use mysqlbinlog To view or to mysql-bin.000007 For example, log , The basic usage is as follows ：

mysqlbinlog "/usr/local/mysql/data/mysql-bin.000007"

mysqlbinlog mysql-bin.000007

mysqlbinlog mysql-bin.000007 > 'test.sql'

Use mysqlbinlog Restore data ：

mysqlbinlog --start-datetime="2022-06-22 11:38:59" --stop-datetime="2022-06-22 11:40:01" 'mysql-bin.000007' | mysql -uroot -proot