Mysql34 other database logs

One . MySQL Supported logs

1.1. Log type

MySQL There are different types of log files , Used to store different types of logs , It is divided into Binary log 、 Error log 、 General query log and Slow query log , This is also commonly used 4 Kind of .MySQL 8 Two more logs are added ： relay logs and Data definition statement log . Use these log files , You can see MySQL What happened inside .
this 6 Class logs are ：

• Slow query log ： Record all execution times over long_query_time Of all inquiries , It is convenient for us to optimize the query .
• General query log ： Record the start time and end time of all connections , And connect all instructions sent to the database server , For the actual scenario of our recovery operation 、 Find the problem , Even the audit of database operation is of great help .
• Error log ： Record MySQL Start of service 、 To run or stop MySQL Problems with service , It is convenient for us to understand the status of the server , So as to maintain the server .
• Binary log ： Record all statements that change data , It can be used for data synchronization between master and slave servers , And the lossless recovery of data in case of server failure .
• relay logs ： Used in master-slave server architecture , An intermediate file used by the slave server to store the binary log contents of the master server . Read the contents of the relay log from the server , To synchronize operations on the primary server .
• Data definition statement log ： Record the metadata operation performed by the data definition statement .
  In addition to binary logs , Other logs are text file . By default , All logs are created in MySQL Data directory in .

1.2. The disadvantages of logging

• The log function will Reduce MySQL Database performance .
• Journal meeting Take up a lot of disk space .

Two . Slow query log (slow query log)

Click to see ：https://blog.csdn.net/qq_43117059/article/details/124914604

3、 ... and . General query log (general query log)

The general query log is used to Record all user actions , Including startup and shutdown MySQL service 、 Connection start time and end time of all users 、 issue MySQL All of the database server SQL Instruction, etc . When our data is abnormal , View the general query log , Specific scenarios during restore operation , It can help us locate the problem accurately .

3.1. View current status

SHOW VARIABLES LIKE '%general%';

3.2. start log :

The way 1： Permanent way :
modify my.cnf perhaps my.ini Configuration file to set . stay [mysqld] Join under group log Options , And restart MySQL service . The format is as follows ：

[mysqld]
general_log=ON
general_log_file=[path[filename]] # Directory path of log file ,filename Is the log file name 

If you do not specify a directory and file name , The general query log will be stored in by default MySQL In the data directory hostname.log In file ,hostname hostname .
The way 2： Temporary way

SET GLOBAL general_log=on; #  Open the general query log 
SET GLOBAL general_log_file=’path/filename’; #  Set the storage location of log files 

Corresponding , Close operation SQL The order is as follows ：

SET GLOBAL general_log=off; #  Close the general query log 

Check the situation after setting ：

SHOW VARIABLES LIKE 'general_log%';

3.3. Check the log

The general query log is based on text file Stored in the file system in the form of , have access to Text editor Directly open the log file . Each station MySQL The general query log content of the server is different .

• stay Windows Operating system , Use the text file viewer ;
• stay Linux In the system , have access to vi Tools or gedit Tool View ;
• stay Mac OSX In the system , You can use a text file viewer or vi Wait for tools to check .
  from SHOW VARIABLES LIKE ‘general_log%’; The location of the general query log can be seen in the results .
  

3.4. Stop logging

The way 1： Permanent way
modify my.cnf perhaps my.ini file , hold [mysqld] Under group general_log Value is set to OFF Or the general_log Comment out one item . After modification and saving , Again restart MySQL service , Effective . give an example 1：

[mysqld]
general_log=OFF

give an example 2：

[mysqld]
#general_log=ON

The way 2： Temporary way
Use SET Statement stop MySQL General query log function ：

SET GLOBAL general_log=off;

Query general log function ：

SHOW VARIABLES LIKE 'general_log%';

3.5. Delete \ Refresh the log

If data is used very frequently , Then the general query log will occupy a very large disk space of the server . The data administrator can delete the query log long ago , In order to make sure MySQL Hard disk space on the server .

3.5.1. Manually delete files

SHOW VARIABLES LIKE 'general_log%';

It can be seen that , The directory of general query log is... By default MySQL Data directory . Manually delete the general query log in this directory .
Use the following command to regenerate the query log file , The specific command is as follows . Refresh MySQL Data directory , Found that a new log file was created . The premise is to open the general log .

mysqladmin -uroot -p flush-logs

Four . Error log (error log)

4.1. start log

stay MySQL In the database , The error log function is Default on Of . and , Error log Cannot be banned .
By default , The error log is stored in MySQL Under the data folder of the database , The name defaults to mysqld.log （Linux System ） or hostname.err （mac System ）. If you need to specify a file name , You need to in my.cnf perhaps my.ini Do the following configuration in ：

[mysqld]
log-error=[path/[filename]] #path Is the directory path where the log file is located ,filename Is the log file name 

After modifying the configuration item , Need to restart MySQL Services to take effect .

4.2. Check the log

MySQL Error logs are stored as text files , You can use a text editor to view .
Query the storage path of the error log ：

SHOW VARIABLES LIKE 'log_err%';

In the execution result, you can see that the error log file is mysqld.log, be located MySQL The default data directory .

4.3. Delete \ Refresh the log

For error logs from a long time ago , Database administrators are unlikely to view these error logs , You can delete these error logs , In order to make sure MySQL On the server Hard disk space .MySQL The error log of is stored in the file system in the form of text file , You can delete .

mysqladmin -uroot -p flush-logs

If appear

mysqladmin: refresh failed; error: 'Could not open file '/var/log/mysqld.log' for
error logging.

Supplementary operation ：

install -omysql -gmysql -m0644 /dev/null /var/log/mysqld.log

5、 ... and . Binary log (bin log)

binlog Can be said to be MySQL Chinese comparison important The Journal of , In the process of daily development and operation and maintenance , Come across .
binlog namely binary log, Binary log file , Also called change log （update log）. It records all the execution of the database DDL and DML Wait for the database update event , However, it does not contain statements that do not modify any data （ Such as data query statement select、show etc. ）.
binlog Main application scenarios ：

• One is for Data recovery
• Second, it is used for Data replication
  

5.1. Check the default

Check whether binary log logging is enabled ： stay MySQL8 By default , Binary files are open .

 show variables like '%log_bin%';

5.2. Log parameter settings

5.2.1. The way 1： Permanent way

modify MySQL Of my.cnf or my.ini Log file related parameters can be set ：

[mysqld]
# Enable binary logging 
log-bin=zyy-bin
binlog_expire_logs_seconds=600
max_binlog_size=100M

Restart MySQL service .
Settings with bin-log Log storage directory ：
If you want to change the directory and name of the log file , It can be done to my.cnf or my.ini Medium log_bin The parameters are modified as follows ：

log-bin="/var/lib/mysql/binlog/zyy-bin"

Restart MySQL service . Query binary log information , Execution results ：

Be careful ： The new folder needs to be used mysql user , Just use the following command .

chown -R -v mysql:mysql binlog

5.2.2. The way 2： Temporary way

If you don't want to set the binary log by modifying the configuration file and restarting , You can also use the following instructions , It is important to note that in mysql8 There are only Session level Set up , period global Level settings .

# global  Level 
mysql> set global sql_log_bin=0;
ERROR 1228 (HY000): Variable 'sql_log_bin' is a SESSION variable and can`t be used
with SET GLOBAL
# session Level 
mysql> SET sql_log_bin=0;
Query OK, 0 rows affected (0.01  second )

5.3. Check the log

When MySQL When creating a binary log file , First create one to “filename” As the name 、 With “.index” A file with a suffix , Create another one to “filename” As the name 、 With “.000001” A file with a suffix .
MySQL service Restart , With “.000001” A suffix will be added to the file , And the suffix name is pressed 1 Increasing . That is, the number of log files is the same as MySQL The service starts the same number of times ; If the log length exceeds max_binlog_size Upper limit （ The default is 1GB）, A new log file will be created .
View the current binary log file list and size . The instructions are as follows ：

SHOW BINARY LOGS;

The following command line event is represented by false SQL In the form of Show it

mysqlbinlog -v "/var/lib/mysql/binlog/zyy-bin.000004"

The previous command also displays binlog Formatted statement , Do not display it using the following command ：

mysqlbinlog -v --base64-output=DECODE-ROWS "/var/lib/mysql/binlog/zyy-bin.000004"

The previous command also displays binlog Formatted statement , Use the following command to not display it about mysqlbinlog There are many skills in using tools , For example, only operations on a library or operations within a certain time period are parsed . Share a few common sentences , For more operations, please refer to the official documents .

#  You can view the parameter help 
mysqlbinlog --no-defaults --help
#  Check out the last 100 That's ok 
mysqlbinlog --no-defaults --base64-output=decode-rows -vv zyy-bin.000004 |tail -100
#  according to position lookup 
mysqlbinlog --no-defaults --base64-output=decode-rows -vv zyy-bin.000004 |grep -A
20 '4939002

Read and take out this way binlog The full-text content of the log is more , It's not easy to distinguish and see pos Some information , Here is a more convenient query command ：

 show binlog events [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count];

• IN ‘log_name’ ： Specify the binlog file name （ No designation is the first binlog file ）.
• FROM pos ： Specify from which pos Start from （ Do not specify is from the entire file first pos Start at ）.
• LIMIT [offset] ： Offset ( No designation is 0).
• row_count : Total number of queries （ Not specifying is all lines ）.

 show binlog events in 'zyy-bin.000004';

What we have said above is based on binlog The default format for ,binlog Format view ：

 show variables like 'binlog_format';

besides ,binlog also 2 format , Namely Row and Mixed

• Statement： Each one will modify the data sql It will be recorded in binlog in .
  advantage ： You don't need to record every line change , Less binlog Log volume , Economize IO, Improve performance .
• Row：5.1.5 Version of MySQL Just started to support row level Copy , It doesn't record sql Statement context information , Save only which record was modified .
  advantage ：row level The log content of will record the details of each row of data modification very clearly . And there will be no stored procedures under certain circumstances , or function, as well as trigger The call and trigger of cannot be copied correctly .
• Mixed： from 5.1.8 Version start ,MySQL Provides Mixed Format , It's actually Statement And Row The combination of .

5.4. Using logs to recover data

mysqlbinlog The syntax for recovering data is as follows ：

mysqlbinlog [option] filename|mysql –uuser -ppass;

This command can be understood as ： Use mysqlbinlog Command to read filename The content in , And then use mysql The command restores these contents to the database .

• filename ： Is the log file name .
• option ： optional , Two more important pairs option Parameter is –start-date、–stop-date and --start-position、–stop-position.

--start-date  and  --stop-date ： You can specify the start time and end time of the recovery point of the database .
--start-position and --stop-position ： You can specify the start and end locations of the recovered data .

Be careful ： Use mysqlbinlog Command to restore , It must be recovered first if the number is small , for example atguigu-bin.000001 Must be in atguigu-bin.000002 Restore before .

5.5. Delete binary log

MySQL The binary files can be configured to be automatically deleted , meanwhile MySQL It also provides a safe way to manually delete binaries .PURGE MASTER LOGS Delete only the specified part of the binary log file , RESET MASTER Delete all binary log files . As follows ：
PURGE MASTER LOGS： Delete the specified log file
PURGE MASTER LOGS The grammar is as follows ：

PURGE {MASTER | BINARY} LOGS TO ‘ Specify the log file name ’
PURGE {MASTER | BINARY} LOGS BEFORE ‘ Specify Date ’

5.6. Other scenes

Binary logs can be accessed through the database Full volume backup And stored in binary logs Incremental information , Complete the database No loss recovery . however , If you encounter a large amount of data 、 There are many databases and data tables （ For example, the application of sub database and sub table ） Scene , Data recovery with binary log , It's very challenging , Because the starting and ending positions are not easy to manage .
under these circumstances , An effective solution is Configure master-slave database server , Even One master, many followers The architecture of , Pass the contents of the binary log file through the relay log , Synchronize to and from the database server , This can effectively avoid data anomalies caused by database failures .

6、 ... and . Let's talk about binary logs (binlog)

6.1. Write mechanism

binlog The timing of writing is also very simple , During transaction execution , First write down the journal binlog cache , When the transaction is submitted , And then binlog cache writes binlog In file . Because of a transaction binlog Can't be taken apart , No matter how big it is , Also make sure to write once , So the system allocates a block of memory for each thread binlog cache.

write and fsync The timing of , It can be determined by parameters sync_binlog control , The default is 0 . by 0 When , Indicates that every time a transaction is committed, only write, It is up to the system to decide when to execute fsync. Although the performance has been improved , But the machine is down ,page cache Inside binglog Will lose . Here's the picture ：

For safety's sake , It can be set to 1 , Indicates that every time a transaction is committed fsync, Just as redo log The process of disc brushing is the same . Finally, there is a compromise , It can be set to N(N>1), Indicates every transaction submitted write, But cumulatively N After a business fsync.

In the presence of IO In the bottleneck scene , take sync_binlog Set to a larger value , Can improve performance . alike , If the machine goes down , Will lose recent N A business binlog journal .

6.2. binlog And redolog contrast

• redo log It is Physical log , The record content is “ What changes have been made on a data page ”, Belong to InnoDB Generated by the storage engine layer .
• and binlog yes Logic log , The record content is the original logic of the statement , Be similar to “ to ID=2 In this line c Field plus 1”, Belong to MySQL Server layer .

6.3. Two-phase commit

During the execution of the update statement , Will record redo log And binlog Two logs , In basic transactions ,redo log During the execution of the transaction, you can continuously write , and binlog Write... Only when the transaction is committed , therefore redo log And binlog Of Write time Dissimilarity .

redo log And binlog The logic between the two logs is inconsistent , What will happen ？

because binlog If you don't finish it, it's abnormal , Now binlog There is no corresponding modification record .

In order to solve the problem of logical consistency between two logs ,InnoDB The storage engine uses a two-phase commit scheme .

After using two-stage submission , write in binlog When an exception occurs, it will not affect

Another scene ,redo log Set up commit There was an exception in the phase , Will the transaction be rolled back ？

The transaction will not be rolled back , It will execute the logic framed in the figure above , although redo log Is in prepare Stage , But through business id Find the corresponding binlog journal , therefore MySQL Think it's complete , The transaction recovery data will be committed .

7、 ... and . relay logs (relay log)

7.1. Introduce

The relay log exists only on the slave server of the master-slave server architecture . In order to be consistent with the master server , To read the contents of the binary log from the master server , And write the read information into Local log files in , The local log file from the server is called relay log . then , Read relay log from server , And update the data from the server according to the contents of the relay log , Complete the of master-slave server Data synchronization .
After setting up the master-slave server , The relay log is saved in the data directory of the slave server by default .
The format of the file name is ： From the server name -relay-bin. Serial number . The relay log also has an index file ： From the server name -relay-bin.index , Used to locate the relay log currently in use .

7.2. Check the relay log

The format of relay log is the same as that of binary log , It can be used mysqlbinlog Tools to view .

SET TIMESTAMP=1618558728/*!*/;
BEGIN
/*!*/;
# at 950
#210416 15:38:48 server id 1 end_log_pos 832 CRC32 0xcc16d651 Table_map:
`MYSQLTEST`.`test` mapped to number 91
# at 1000
#210416 15:38:48 server id 1 end_log_pos 872 CRC32 0x07e4047c Delete_rows: table id
91 flags: STMT_END_F -- server id 1  Is the primary server , It means that the main server deleted a line of data 
BINLOG '
CD95YBMBAAAAMgAAAEADAAAAAFsAAAAAAAEABGRlbW8ABHRlc3QAAQMAAQEBAFHWFsw=
CD95YCABAAAAKAAAAGgDAAAAAFsAAAAAAAEAAgAB/wABAAAAfATkBw==
'/*!*/;
# at 1040

This passage means , master server （“server id 1”） Counter table MYSQLTEST.test the 2 Step by step ：

 Go to the table  MYSQLTEST.test  The number is  91  The record of , The log location is  832;
 Delete number is  91  The record of , The log location is  872.

7.3. Typical error of recovery

If the slave server goes down , Sometimes for system recovery , To reinstall the operating system , This may lead to your Server name Prior to Different . And the relay log is Contains the server name Of . under these circumstances , It may cause you to recover from the server , Unable to read data from the relay log before downtime , Thought the log file was corrupted , In fact, the name is wrong .
The solution is simple , Change the name from the server back to the previous name .