Zero copy technology of MySQL

First you need to understand Buffer And cache The difference between

Bbuffer And Cache Very similar , Because they are used to store data , Read byte data by the application layer . They have the same concept on many occasions :

First, from the translation ,Buffer Should be translated as “ buffer ”,Cache Should be translated as “ cache ”, Two are not the same thing at all .

At the hardware level ,Buffer Expected memory ,Cache by CPU Integrated tell cache .

Buffer In order to synchronize devices with different speeds , Create a buffer area , Write in Buffer The data is to be taken out and written to other devices .

Cache To improve the reading speed , Pre read the frequently or immediately needed data into the cache , Write in Cache The data is read by other devices .

From the software layer ,Buffer It is the buffer of the block device ,Cache It's the file system cache . With Linux For example ,

Buffer(Buffer Cache) Buffering the operation of block devices in block form , Timing or manual synchronization to the hard disk , It's to buffer write operations and then write many changes to the hard disk at one time , Avoid frequent hard disk writing , Improve write efficiency .

Cache(Page Cache) File system files are cached in the form of pages , Read for the program you need to use , It's to provide a buffer for read operations , Avoid frequent hard disk reading , Improve reading efficiency .

To make a long story short ,Buffer The contents are for writing elsewhere ,Cache The contents are meant to be read elsewhere .

Buffer And Cache The use of ：

Buffer The main purpose of is in different applications 、 Threads 、 Shared byte data between processes , For example, in order to enable devices with different speeds to synchronize data , Will use sharing Buffer;

Cache The main purpose of is to improve the reading of byte data / Write speed , For example, according to the time locality 、 The address locality operating system provides page cache Mechanism ;

Of course , On many occasions Buffer And Cache Have the same semantics , Therefore, we can think that the buffer is used to improve the reading and writing speed , It is also used for data sharing and synchronization .

2. MySQL Buffer design

MySQL The buffer design of is shown in the following figure ：

Figure1.MySQL Buffer design

As shown in the figure above ,MySQL Different supporting technologies different from caching mechanism are used at different levels . Among them is ：

application layer ：

Redo Log Buffer： Cache writes , Used to implement MySQL InnoDB The transactional nature of ;

InnoDB Buffer Pool： Used to deal with MySQL table Data for caching . Read memory instead of disk , Improve the performance of read operation by reducing disk read operation ; Write memory instead of disk , Improve write performance by reducing disk write operations ;

Operating system VFS（Virtual file system, Virtual file system ） layer ：

Page Cache： The operating system controls the data in the file system through caching and read ahead mechanism block be based on page Cache management ;

Direct Buffer： When using Direct I/O Related to the offer API when , The operating system no longer provides... Based Page Cache Caching mechanism , But directly Direct Buffer;

On disk Disk Buffer： Disk can also provide disk cache , Usually in MySQL The disk cache will be turned off in , We just need to understand that there are Disk Buffer This concept is sufficient .

3. Write Through/Back And Direct I/O

Write Through And Write Back It refers to whether the application using memory space as cache will drop directly when processing write operations ：

Write Through： Write operations " Through the " Directly drop the cache , This strategy can ensure that the data in the memory buffer will not be lost due to downtime ;

Write Back： A write operation only updates the data in the memory cache , Data dropping is usually done once every other time ;

MySQL Some parameters are provided to control Page Cache The specific behavior of data dropping , for example ：

（1）innodb_flush_log_at_trx_commit

innodb_flush_log_at_trx_commit The parameter is used to control based on Page Cache Of Redo Log Buffer Data drop mechanism [2]. This parameter is used to control the balance between the following two features ：

Strict transaction management mechanism ;

Transaction submission commit High performance during operation execution ;

innodb_flush_log_at_trx_commit There are three optional configuration values ：

1（ The default value is ）： Every time a transaction is committed, the log must be flushed to disk , Provides the most reliable transactional assurance ;

0： Log every interval 1 Seconds to disk , This means that data in the cache that has not been refreshed to disk in time will be lost in case of downtime ;

2： Log after transaction commit and every interval 1 Seconds to disk , This means that data in the cache that has not been refreshed to disk in time will be lost in case of downtime ;

matters needing attention ： To configure 0 And 2 There's no guarantee 100% Refresh to disk every second , This is because DDL Modification of and InnoDB Activity may cause logs to refresh more frequently . On the other hand , Due to the transaction scheduling problem , The refresh rate will even decrease .

The default refresh rate is 1 s, By the parameter innodb_flush_log_at_timeout To configure .

（2）innodb_flush_method

innodb_flush_method The parameters are controlled at the same time redo log buffer and innodb buffer pool Buffer refresh policy , among ：

log files：redo log buffer yes log files Cache in memory , log files It's on disk Redo Log file ;

data files：innodb buffer pool yes data files Cache in memory ,data files It's a data file on disk （B+tree）;

innodb_flush_method Parameters currently have 6 Optional configuration values [3]：

fdatasync;

O_DSYNC

O_DIRECT

O_DIRECT_NO_FSYNC

littlesync

nosync

This is only about Unix-like operating system , Without discussion Windows System .

among ,littlesync And nosync For internal performance testing only , Not recommended .

fdatasync, That is, the value 0, This is the default configuration value . Yes log files as well as data files All use fsync Synchronization by ;

O_DSYNC, That is, the value 1. Yes log files Use O_SYNC Open and refresh log files , Use fsync To refresh data files Data in ;

O_DIRECT, That is, the value 4. utilize Direct I/O The way to open data file, Write and execute every operation fsync Drop the disk in the way of system call ;

O_DIRECT_NO_FSYNC, That is, the value 5. utilize Direct I/O The way to open data files, But each write operation does not call fsync The system calls to drop the disk ;

Additional explanation ： With O_SYNC Opening a file in mode means that every write operation of the file directly causes the data itself and metadata to be refreshed to the disk .

Why O_DIRECT And O_DIRECT_NO_FSYNC Configuration difference ？

First , We need to understand that the update operation is divided into two specific sub steps ：① File data update drop disk ② File metadata update drop disk .O_DIRECT In some operating systems, the file metadata will not fall off the disk , Unless you call fsync, So ,MySQL Provides O_DIRECT as well as O_DIRECT_NO_FSYNC These two configurations [5].

If you are sure you are on your own operating system , Even if not fsync call , It can also ensure that the file metadata falls to the disk , Then please use O_DIRECT_NO_FSYNC To configure , This is right MySQL Performance helps a little . otherwise , Please use O_DIRECT, Otherwise, the loss of file metadata may lead to MySQL Running error .

4. MySQL Log refresh strategy

MySQL The log refresh policy passes sync_binlog Parameters to configure , Its have 3 Optional configuration ：

sync_binlog=0：MySQL The application will not be responsible for log synchronization to disk at all , Refresh the log data in the cache to the disk and leave it to the operating system to complete ;

sync_binlog=1：MySQL The application flushes the log of the cache to disk before the transaction is committed ;

sync_binlog=N： When N Not for 0 And 1 when ,MySQL In the collection N After a log is submitted , Will synchronize the logs of the cache to the disk .

in fact , This parameter is also used to control the logging through Write Through still Write Back Policy flushes to disk .

matters needing attention ： Use Page Cache Data disk brushing mechanism of mechanism , Even based on synchronization policy , That is, each write operation requires data to be directly dropped onto the disk , But before the data falls , Data is always written in Page Cache in , then Page Cache The specific Page Refresh to disk .

5. MySQL Typical configuration of

innodb_flush_log_at_trx_commit Parameter is configured as 1：Redo Log go Page Cache, And the log of each write operation passes before the transaction is committed fsync Refresh to disk ;

innodb_flush_method Parameter is configured as O_DIRECT：InnoDB Buffer Pool go Direct I/O, And the file data caused by each write operation （ Including file metadata ） All pass fsync System call refresh to disk ;

Write a redo log The steps involved are ：

Log write Redo Log buffer;

Log write Page Cache;

By system call fsync take Page Cache The dirty pages in are flushed to disk ;

Log submission ;

The steps involved in modifying a row of records in a table are ：

The updated data is written in InnoDB Buffer Pool;

The following logic is performed regularly （ asynchronous ）：

InnoDB Buffer Pool Refresh dirty data , Through documents write methods ;

Of documents write Method directly causes data to be written to disk ;

Regular file fysnc call , Make sure that the file metadata is written on disk ;

REFERENCE
[1]Buffer And Cache

[2]MySQL :: MySQL 8.0 Reference Manual :: 15.14 InnoDB Startup Options and System Variables

[3]MySQL 8.0 innodb_flush_method

[4]MySQL :: MySQL 8.0 Reference Manual :: 17.1.6.4 Binary Logging Options and Variables

[5] Why MYSQL still use fsync() to flush the data when the option is O_DIRECT?

The source code attachment has been packaged and uploaded to Baidu cloud , You can download it yourself ～

link : https://pan.baidu.com/s/14G-bpVthImHD4eosZUNSFA?pwd=yu27
Extraction code : yu27
Baidu cloud link is unstable , It may fail at any time , Let's keep it tight .

If Baidu cloud link fails , Please leave me a message , When I see it, I will update it in time ～

Open source address
Code cloud address ：
http://github.crmeb.net/u/defu

Github Address ：
http://github.crmeb.net/u/defu

link ：https://spongecaptain.cool/post/mysql/zerocopyofmysql