[MySQL from introduction to proficiency] [advanced chapter] (V) SQL statement execution process of MySQL

Hello! , I'm Manon Feige , Thank you for reading this article , Welcome to three links with one button .
1. Python Basic column , Basic knowledge in a net ,9.9 Yuan can't buy a loss , I can't buy it . Python From entry to mastery
️ 2. Python Crawler column , Systematically learn the knowledge points of reptiles .9.9 Yuan can't buy a loss , I can't buy it .python Reptile beginner level
️ 3. Ceph actual combat , Everything from principle to actual combat . Ceph actual combat
️ 4. Java Introduction to high concurrency programming , Punch in to learn Java High concurrency . Java Introduction to high concurrency programming
5. Take a stroll around the community , Weekly benefits , There are surprises every week . Manon Feige community , Leap plan
The whole network has the same name 【 Manon Feige 】 Welcome to your attention , personal VX: wei158556

1. brief introduction

Today, we officially enter the study of core knowledge points , This paper mainly introduces MySQL in SQL The implementation process of . be familiar with SQL The implementation process of is very important for the later optimization of the database . Don't talk too much, go straight to today's study .

2. Environmental Science

Environmental Science	edition
Red Hat	4.8.5-39
MySQL	5.7

SQL Execute the process

SQL Statements in MySQL The execution process in , It can be roughly divided into ：

The client will send a SQL Statement passed to MySQL After the server , If the query cache setting is turned on , First query the cache , If the view cache is not opened or the cache misses, then the parser will parse the SQL sentence , After the parsing is completed, the optimizer pair SQL Statement optimization , After optimization, the actuator executes the SQL sentence , And return the execution result to the client .

Detailed implementation process

SQL The detailed execution process of the statement is shown in the following figure ：

1. The query cache

（ Enable query cache settings ,MySQL5.7 Not on by default ,MySQL8.0）：
Server If you find this in the query cache SQL sentence , The result will be returned directly to the client ; without , We're in the parser phase . It should be noted that , Because query caching is often inefficient （ The hit rate is not high ）, So in MySQL8.0 After that, I abandoned this function . in addition ,MySQL5.7 Query caching is not enabled by default .

MySQL After getting a query request , I will go to query cache first , Have you executed this statement before , Previously executed statements and results may be in the form of key-value On the form of , Is directly cached in memory .key Is the statement of the query ,value Is the result of a query , If your query can be found directly in this cache key, So this value Will be returned directly to the client . If the query statement is not in the query cache , I'm going to continue the execution phase , After execution , The execution result will be stored in the query cache . therefore , If you query the command cache ,MySQL There is no need to perform the following complex operations , I can just return the result , It's very efficient .
however ..........
Most of the time , Query cache is a chicken rib , There are several reasons ：

• The query hit rate is not high ,
  stay MySQL Query caching in , Not a cached query plan , Instead, query the corresponding results , Only The same query operation will command the query cache . The difference between two query requests in any character （ for example ： Space 、 notes 、 Case write ）, Will cause the cache not to be hit , therefore MySQL The query cache hit rate of is not high .
  If the query request contains some system functions 、 User defined functions and functions , such as NOW() function , Each call will generate the current time of the crime , Cannot hit the query cache every time .
• Query expiration time is short
  If the table is frequently modified , The query cache is also frequently modified , This will increase the consumption of the system , Cause frequent cache invalidation .
  All in all , Because query caching does more harm than good , Query cache failures are very frequent .
  It is generally recommended that you use query caching in static tables , The so-called static table , Data tables that are rarely updated , such as ： Configuration table of a system , Dictionary table , The query on this table is suitable for query caching .

Check whether the query cache is enabled

stay MySQL 5.7 View the current MySQL Whether the instance enables the caching mechanism

show variables like 'query_cache_type';

It is not turned on by default , How to enable query caching will be described in detail later .query_cache_type There are three values ,0 On behalf of closing the query cache OFF,1 Representative opens ON,2 Indicates on demand , If set to 2 Words , When you need to use the query cache , have access to SQL_CACHE, Like the following statement ：

SELECT SQL_CACHE * FROM test WHERE ID=10;

Monitor the hit rate of the query cache

You can monitor the hit rate of query cache through the following command ：

show status like '%Qcache%';

Analysis of running results ：

1. Qcache_free_blocks: Indicates how much is left in the query cache blocks, If the value is large , This indicates that there are too many memory fragments in the query cache , It may be organized at a certain time .
2. Qcache_free_memory: Query the memory size of the cache , Through this parameter, you can clearly guide whether the current system has enough query memory , Are there , Still not enough ,DBA It can be adjusted according to the actual situation .
3. Qcache_hits: Indicates how many command caches there are , We can mainly verify the effect of our query cache by this value , The greater the number , The better the cache .

2. Parser

In the parser SQL Statement syntax analysis , Semantic analysis .

If there is no command to query the cache , It's about time to actually execute the statement . First ,MySQL Need to know what you're going to do , So you need to SQL Statement parsing ,SQL Sentence analysis is divided into lexical analysis and grammatical analysis .
First do **“ Lexical analysis ”**, What you enter is a string with multiple Spaces SQL sentence ,MySQL You need to identify the strings in it , What is the .

MySQL From what you typed SELECT This keyword recognizes , This is a query statement , It also takes strings "T" Identify a " Table name T", Put the string "ID" Identify a " Column ID".
next , Want to do " Syntax analysis ". According to the result of lexical analysis , parsers （ such as ：Bison） According to the rules of grammar , Judge the one you typed SQL Does the statement satisfy SQL grammar .
If your statement is wrong , Will receive "You have an error in your SQL syntax" Error prompt for , Take the following statement "from" It has been written. “fro”.

mysql> SELECT * FRO test WHERE ID=1;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'FRO test WHERE ID=1' at line 1

If SQL The statement is correct , A syntax tree like this will be generated ：
As follows SQL To illustrate ：

SELECT username,age FROM userinfo WHERE age>20 and level>5;

Here is SQL The process and steps of lexical analysis ：

The basic work of the parser is lexical analysis and grammatical analysis , Reach a syntax tree , Then give it to its downstream .

3. Optimizer

It will be determined in the optimizer that SQL Statement execution path , For example, according to Full search , Or according to Index retrieval, etc .
After the parser ,MySQL I know what you want to do , Before we start executing , It's also handled by the optimizer , A query can be executed in many ways , All return the same result . The role of the optimizer is to find the best execution plan .

such as ： The optimizer is when there are multiple indexes in the table , Decide which index to use ; Or there are multiple table associations in a statement （join） To determine the connection order of each table , And expression simplification 、 Turn subquery into connection , From external connection to internal connection .
give an example ： The following statement executes two tables join;

  SELECT * FROM test1 join test2 using(ID)
  WHERE test1.name='zhangsan' and test2.name='MySQL From introductory to proficient courses ';

Scheme 1 ： You can start from the table test1 The inside out name=‘zhangsan’ The record of ID value , On the basis of ID Value association table test2, In judging test2 Inside name Is the value of ’MySQL From introductory to proficient courses ’.

Option two ： You can start from the table test2 The inside out name='MySQL From introductory to proficient courses ’ The record of ID value , According to ID The value associated with the test1, To determine test1 Inside name Is the value of zhangsan.

The logical result of the two execution methods is the same , But the efficiency of implementation will be different , The role of the optimizer is to decide which scheme to use , After the optimizer phase is complete , The execution of this statement is determined , Then we go to the executor phase .

In the query optimizer , It can be divided into logical query optimization stage and physical query optimization stage .

Logical query optimization is by changing SQL The content of the statement to make SQL Queries are more efficient , At the same time, more candidate execution plans are provided for physical query optimization . The usual way is to SQL Statement , Rewrite the query , The mathematical basis of query rewriting is relational algebra . Equivalent predicate rewriting of conditional expressions 、 Condition simplification , Rewrite the view , Optimize subqueries , The connection semantics is eliminated by external connection , Nested connection elimination, etc .

Physical query optimization is query rewriting based on relational algebra , Every step of relational algebra corresponds to physical calculation , There are often many algorithms for these physical calculations , Therefore, we need to calculate the cost of various physical paths , So choose the one with the least cost as the implementation plan . At this stage , For the operation of single table and multi table connection , Need to use indexes efficiently , Improve query efficiency .

4. actuator ：

Up to now , I haven't really read and write real tables yet , It just produced an implementation plan , So it goes into the actuator stage .
Before execution, you need to judge whether the user has permission , without , A permission error will be returned . If you have permission , Is executed SQL Query and return the result . stay MySQL8.0 The following version , If query cache is set , At this time, the query results will be cached .

SELECT * FROM test WHERE id=1;

If you have authority , Open the table and continue . When I open my watch , The actuator is defined according to the engine of the table , Call the storage engine API Read and write the table . Storage engine API Just an abstract interface , There is also a storage engine layer below , It depends on the implementation of the storage engine .

such as ： surface test in ,ID Field has no index , So the execution process of the actuator is like this ：
call InnoDB The engine interface takes the first row of this table , Judge ID Value is 1, If not, skip , If it is, the row will exist in the result set ; Call the engine interface " The next line ", Repeat the same logic of judgment , Until you get to the last row of the table .

The executor returns the record set composed of all the rows that meet the conditions in the traversal process to the client as a result set .

thus , This statement is executed , For tables with indexes , The execution logic is similar .

SQL Demonstration of execution process

Since one SQL Statements go through different modules , Let's take a look at , Under different modules ,SQL Resources used for execution （ Time ） What is the . How to be in MySQL Middle to one SQL The execution time of the statement is analyzed .

1. confirm profiling Open or not

Understand the underlying execution process of query statements ：select @@profiling; perhaps show variables like '%profiling%' Check whether the plan is enabled . Turn it on so that MySQL Collected in SQL Resources used in execution , The order is as follows ：

#  The default query is session Grade , If you want to query the global level, you can use  select @@global.profiling;
mysql> select @@profiling;
mysql> show variables like 'profiling';

profiling=0 On behalf of closed , We need to take profiling open , It is set to 1;

mysql> set profiling=1;

Profiling Function by MySQL Session variables ：profiling control , The default is OFF（ The closed position ）.

2. Perform the same many times SQL Inquire about

Then we execute either SQL Inquire about .

SELECT * FROM customer limit 100;

Next, check each... With the following command SQL Execution time of .

mysql> show profiles;

If you want to view the latest SQL The specific execution plan of the statement can be achieved through the following commands ;

mysql> show profile;

Here is a detailed display of SQL sentence SELECT * FROM customer limit 100; The execution of each step takes time , Include start , Check authority , Open the data table , initialization , System lock, etc .
If you want to query a specific item SQL The detailed implementation of can be achieved through the following commands ;

mysql> show profile for query [Query_ID];

Such as query Query_ID by 4 Of SQL Statement execution .

Turn on query cache

1. Open the query cache in the configuration file
stay /etc/my.cnf Add a new line ：

query_cache_type=1

2. restart mysql service

systemctl restart mysqld

3. Open the query execution plan
Because the service has been restarted , The following instructions need to be executed again , Turn on profiling.

mysql> set profiling=1;

4. Execute the statement twice ：

mysql>SELECT * FROM stock LIMIT 100;

mysql>SELECT * FROM stock LIMIT 100;

Query after execution SQL The total time of the statement , It is found that the total time of the second execution is significantly shorter than that of the first execution .

Check separately SQL The time-consuming of statements in each link , Find the first execution SQL There are 29 Status

In the second execution, only 9 Status , And you can see sending cached result to client This state , so , The second query is to return data directly from the query cache .

Of course ,profile Commands take time apart from querying , You can also query the cost of each stage cpu,io The situation of .

It should be noted that SQL Statements must be completely consistent to hit the cache , Otherwise, the cache cannot be hit .
Now modify the above query statement slightly ;

SELECT * from  stock LIMIT 100;

take form Change to lowercase , stay from and stock Add space between , It is found that the cache cannot be hit .

summary

Have a good command of SQL The implementation process of is very, very important ！！
This article is mainly about three points ;

1. It is not recommended to enable the query cache
2. SQL The implementation process of is mainly ：SQL sentence –> The query cache –> Parser —> Optimizer —> actuator
3. adopt select profiles You can query all sql The execution time of , adopt select profile You can query the latest sql The time-consuming of each stage of