MySQL advanced part 2: optimizing SQL steps

In the process of application development , Due to the small amount of data in the initial stage , Developers write SQL We should pay more attention to the realization of function when we make a statement , But when the application system goes online , With the rapid growth of production data , quite a lot SQL Statements begin to show performance problems , The impact on production is also growing , At this point, these problems SQL Statement becomes the bottleneck of the whole system performance , So we have to optimize them , This chapter will introduce in detail in MySQL Medium optimization SQL Method of statement .

When faced with a person who has SQL Performance issues with the database , Where should we start to make a systematic analysis , Enables the problem to be located as soon as possible SQL And solve the problem as soon as possible .

see SQL Frequency of execution

MySQL After successful client connection , adopt show [session|global] status Command can provide server status information .show [session|global] status You can add parameters as needed “session” perhaps “global” To display session level （ Current connection ） Statistical results and global level （ Since the database was last started ） The statistics of . If you don't write , The default usage parameter is “session”.

The following command shows the current session The values of all statistical parameters in ：

show status like 'Com_______';

show status like 'Innodb_rows_%';

Com_xxx Represent each xxx Number of statement executions , We are usually concerned with the following statistical parameters .

Parameters	meaning
Com_select	perform select Number of operations , A query only adds up to 1.
Com_insert	perform INSERT Number of operations , For batch inserted INSERT operation , Only add up once .
Com_update	perform UPDATE Number of operations .
Com_delete	perform DELETE Number of operations .
Innodb_rows_read	select The number of rows returned by the query .
Innodb_rows_inserted	perform INSERT The number of rows inserted by the operation .
Innodb_rows_updated	perform UPDATE Number of rows updated by operation .
Innodb_rows_deleted	perform DELETE The number of rows deleted by the operation .
Connections	Trying to connect MySQL The number of servers .
Uptime	Server working hours .
Slow_queries	The number of slow queries .

Com_*** : These parameters are accumulated for all table operations of the storage engine .

Innodb_*** : These parameters are only for InnoDB Storage engine , The algorithm of accumulation is slightly different .

Positioning inefficient execution SQL

There are two ways to locate the less efficient SQL sentence .

• Slow query log : Slow query logs to locate those that are less efficient SQL sentence , use –log-slow-queries[=file_name] When the option starts ,mysqld Write a containing all execution times over long_query_time Of a second SQL Statement log file . For details, please refer to chapter 26 The related parts of log management in Chapter 2 .

• show processlist : Slow query logs are only recorded after the query is finished , So when the application reflects the problem of execution efficiency, the query of slow query log cannot locate the problem , have access to show processlist Command to view the current MySQL Thread in progress , Including the state of the thread 、 Lock the watch, etc , Can view... In real time SQL Implementation of , At the same time, some lock table operations are optimized .

  ( For example, a new client (sqlyog/navicat) Connect current mysql service , There will be one more item in the table below )

1） id Column , The user login mysql when , System assigned "connection_id", You can use functions connection_id() see 

2） user Column , Show current user . If not root, This command only displays the scope of user authority sql sentence 

3） host Column , Show from which ip On which port of , It can be used to track users who have problem statements 

4） db Column , Shows which database the process is currently connected to 

5） command Column , Displays the command executed by the current connection , Generally, the value is sleep （sleep）, Inquire about （query）, Connect （connect） etc. 

6） time Column , Shows the duration of this state , The unit is seconds 

7） state Column , Shows the... Using the current connection sql The state of the statement , A very important column .state Describes a state in the execution of a statement . One sql sentence , Take the query as an example , May need to go through copying to tmp table、sorting result、sending data Wait for the status to complete 

8） info Column , Show this sql sentence , It is an important basis to judge the problem statement 

explain Analysis execution plan

Through the above steps to find out the inefficient SQL After the statement , Can pass EXPLAIN perhaps DESC Command acquisition MySQL How to execute SELECT Statement information , Included in SELECT How tables are joined and the order in which they are joined during statement execution

Inquire about SQL Statement execution plan ：

explain  select * from tb_item where id = 1;

explain  select * from tb_item where title = ' Alcatel  (OT-979)  The glacier white   Unicom 3G mobile phone 3';

Field	meaning
id	select The serial number of the query , It's a set of numbers , Represents execution in query select Clause or the order of the operation table .
select_type	Express SELECT The type of , Common values are SIMPLE（ A simple watch , That is, no table join or subquery is used ）、PRIMARY（ Main query , That is, the outer query ）、UNION（UNION The second or subsequent query statement in ）、SUBQUERY（ First in subquery SELECT） etc.
table	Output result set table
type	Indicates the connection type of the table , The connection types with good to poor performance are ( system —> const -----> eq_ref ------> ref -------> ref_or_null----> index_merge —> index_subquery -----> range -----> index ------> all )
possible_keys	When representing a query , Possible indexes
key	Indicates the index actually used
key_len	Length of index field
rows	Number of scan lines
extra	Description and description of the implementation

Environmental preparation

CREATE TABLE `t_role` (
  `id` varchar(32) NOT NULL,
  `role_name` varchar(255) DEFAULT NULL,
  `role_code` varchar(255) DEFAULT NULL,
  `description` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `unique_role_name` (`role_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


CREATE TABLE `t_user` (
  `id` varchar(32) NOT NULL,
  `username` varchar(45) NOT NULL,
  `password` varchar(96) NOT NULL,
  `name` varchar(45) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `unique_user_username` (`username`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


CREATE TABLE `user_role` (
  `id` int(11) NOT NULL auto_increment ,
  `user_id` varchar(32) DEFAULT NULL,
  `role_id` varchar(32) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `fk_ur_user_id` (`user_id`),
  KEY `fk_ur_role_id` (`role_id`),
  CONSTRAINT `fk_ur_role_id` FOREIGN KEY (`role_id`) REFERENCES `t_role` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
  CONSTRAINT `fk_ur_user_id` FOREIGN KEY (`user_id`) REFERENCES `t_user` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;




insert into `t_user` (`id`, `username`, `password`, `name`) values('1','super','$2a$10$TJ4TmCdK.X4wv/tCqHW14.w70U3CC33CeVncD3SLmyMXMknstqKRe',' Super administrator ');
insert into `t_user` (`id`, `username`, `password`, `name`) values('2','admin','$2a$10$TJ4TmCdK.X4wv/tCqHW14.w70U3CC33CeVncD3SLmyMXMknstqKRe',' System administrator ');
insert into `t_user` (`id`, `username`, `password`, `name`) values('3','itcast','$2a$10$8qmaHgUFUAmPR5pOuWhYWOr291WJYjHelUlYn07k5ELF8ZCrW0Cui','test02');
insert into `t_user` (`id`, `username`, `password`, `name`) values('4','stu1','$2a$10$pLtt2KDAFpwTWLjNsmTEi.oU1yOZyIn9XkziK/y/spH5rftCpUMZa',' Student 1');
insert into `t_user` (`id`, `username`, `password`, `name`) values('5','stu2','$2a$10$nxPKkYSez7uz2YQYUnwhR.z57km3yqKn3Hr/p1FR6ZKgc18u.Tvqm',' Student 2');
insert into `t_user` (`id`, `username`, `password`, `name`) values('6','t1','$2a$10$TJ4TmCdK.X4wv/tCqHW14.w70U3CC33CeVncD3SLmyMXMknstqKRe',' teacher 1');



INSERT INTO `t_role` (`id`, `role_name`, `role_code`, `description`) VALUES('5',' Student ','student',' Student ');
INSERT INTO `t_role` (`id`, `role_name`, `role_code`, `description`) VALUES('7',' teacher ','teacher',' teacher ');
INSERT INTO `t_role` (`id`, `role_name`, `role_code`, `description`) VALUES('8',' Teaching administrator ','teachmanager',' Teaching administrator ');
INSERT INTO `t_role` (`id`, `role_name`, `role_code`, `description`) VALUES('9',' Administrators ','admin',' Administrators ');
INSERT INTO `t_role` (`id`, `role_name`, `role_code`, `description`) VALUES('10',' Super administrator ','super',' Super administrator ');


INSERT INTO user_role(id,user_id,role_id) VALUES(NULL, '1', '5'),(NULL, '1', '7'),(NULL, '2', '8'),(NULL, '3', '9'),(NULL, '4', '8'),(NULL, '5', '10') ;

explain And id

id The fields are select The serial number of the query , It's a set of numbers , Represents execution in query select Clause or the order of the operation table .id There are three situations ：

1） id The same means that the order in which tables are loaded is from top to bottom .

explain select * from t_role r, t_user u, user_role ur where r.id = ur.role_id and u.id = ur.user_id ;

2） id Different id The bigger the value is. , The higher the priority , The first to be executed .

EXPLAIN SELECT * FROM t_role WHERE id = (SELECT role_id FROM user_role WHERE user_id = (SELECT id FROM t_user WHERE username = 'stu1'));

3） id It's the same , There are different , At the same time .id The same can be thought of as a group , From top to bottom ; In all groups ,id The greater the value of , The higher the priority , Execute first .

EXPLAIN SELECT * FROM t_role r , (SELECT * FROM user_role ur WHERE ur.`user_id` = '2') a WHERE r.id = a.role_id ; 

explain And select_type

Express SELECT The type of , Common values , As shown in the following table ： From top to bottom, efficiency is getting lower and lower

select_type	meaning
SIMPLE	ordinary select Inquire about , The query does not contain subqueries or UNION
PRIMARY	If the query contains any complex subqueries , The outermost query is marked with this identifier
SUBQUERY	stay SELECT or WHERE The list contains subqueries
DERIVED	stay FROM Subqueries included in the list , Marked as DERIVED（ derivative ） MYSQL These subqueries will be executed recursively , Put the results in the provisional table
UNION	If the second SELECT Appear in the UNION after , Then it is marked with UNION ; if UNION Included in FROM Clause , Outer layer SELECT Will be marked as ： DERIVED
UNION RESULT	from UNION Table to get the result SELECT

explain And table

Show which table this row of data is about

explain And type

type It shows the type of access , Is a more important indicator , It can be taken as ：

type	meaning
NULL	MySQL Don't access any tables , Indexes , Direct return
system	There is only one line in the table ( It's equal to the system table ), This is a const Special case of type , In general, it will not appear
const	Indicates that it is found through index once ,const For comparison primary key perhaps unique Indexes . Because only one line of data is matched , So soon . For example, place the primary key in where In the list ,MySQL You can convert the query to a constant .const At will " Primary key " or " only " All parts of the index are compared with constant values
eq_ref	similar ref, The difference is that you use a unique index , Use primary key Association query , There is only one record found by association . Common in primary key or unique index scan
ref	Non unique index scan , Returns all rows that match a single value . In essence, it is also an index access , Returns all rows that match a single value （ Multiple ）
range	Retrieve only rows returned for a given , Use an index to select rows . where After that between , < , > , in Wait for the operation .
index	index And ALL The difference is index Type just traverses the index tree , Often than ALL fast , ALL It's traversing data files .
all	Will traverse the entire table to find the matching rows

The result is from the best to the worst ：

NULL > system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL


system > const > eq_ref > ref > range > index > ALL

Generally speaking , We need to make sure that the query is at least range Level , It's best to achieve ref .

explain And key

possible_keys :  Show the indexes that may be applied to this table ,  One or more . 

key ：  Actual index used ,  If NULL,  No index is used .

key_len :  Represents the number of bytes used in the index ,  This value is the maximum possible length of the index field , It's not the actual length , Without losing accuracy ,  The shorter the length, the better  .

explain And rows

Number of scan lines .

explain And extra

Other additional execution plan information , Show in this column . The first two that need to be optimized , Keep the back using index

extra	meaning
using filesort	explain mysql Will use an external index to sort the data , Instead of reading in the order of the indexes in the table , be called “ File sorting ”, Low efficiency .
using temporary	Temporary tables are used to save intermediate results ,MySQL Use temporary tables when sorting query results . Common in order by and group by; Low efficiency
using index	It means corresponding select The operation uses an override index , Avoid accessing table data rows , Good efficiency .

show profile analysis SQL

Mysql from 5.0.37 The version began to add right show profiles and show profile Statement support .show profiles Be able to do SQL Optimization helps us understand where the time is spent .

adopt have_profiling Parameters , Be able to see the present MySQL Do you support profile：

Default profiling Is turned off , Can pass set Statements in Session Level on profiling：

set profiling=1; // Turn on profiling  switch ;

adopt profile, We can understand more clearly SQL The process of execution .

First , We can perform a series of operations , As shown in the figure below ：

show databases;

use db01;

show tables;

select * from tb_item where id < 5;

select count(*) from tb_item;

After executing the above order , Re execution show profiles Instructions , Check it out. SQL Statement execution time ：

adopt show profile for query query_id You can see the SQL The state and time consumed by each thread during execution ：

TIP ：
	Sending data  State means MySQL The thread starts to access the data row and returns the result to the client , Instead of just returning a client . Because in Sending data State, ,MySQL Threads often need to do a lot of disk read operations , So it is often the most time-consuming state in the whole query .

After getting the most time consuming thread state ,MySQL Support further choice all、cpu、block io 、context switch、page faults And other detail type classes MySQL Spend too much time on what resources to use . for example , Select View CPU Time consuming ：

Field	meaning
Status	sql The state of statement execution
Duration	sql The time of each step in the execution process
CPU_user	The current user has cpu
CPU_system	System owned cpu

trace Analyze optimizer execution plan

MySQL5.6 Provide for the right to SQL Keep track of trace, adopt trace File to learn more about why the optimizer chose A plan , Instead of choosing B plan .

open trace , Format as JSON, And set up trace Maximum available memory size , Avoid that the default memory is too small to be fully displayed during parsing .

SET optimizer_trace="enabled=on",end_markers_in_json=on;
set optimizer_trace_max_mem_size=1000000;

perform SQL sentence ：

select * from tb_item where id < 4;

Last , Check information_schema.optimizer_trace We can know MySQL How to execute SQL Of ：

select * from information_schema.optimizer_trace\G;
*************************** 1. row ***************************
QUERY: select * from tb_item where id < 4
TRACE: {
  "steps": [
    {
      "join_preparation": {
        "select#": 1,
        "steps": [
          {
            "expanded_query": "/* select#1 */ select `tb_item`.`id` AS `id`,`tb_item`.`title` AS `title`,`tb_item`.`price` AS `price`,`tb_item`.`num` AS `num`,`tb_item`.`categoryid` AS `categoryid`,`tb_item`.`status` AS `status`,`tb_item`.`sellerid` AS `sellerid`,`tb_item`.`createtime` AS `createtime`,`tb_item`.`updatetime` AS `updatetime` from `tb_item` where (`tb_item`.`id` < 4)"
          }
        ] /* steps */
      } /* join_preparation */
    },
    {
      "join_optimization": {
        "select#": 1,
        "steps": [
          {
            "condition_processing": {
              "condition": "WHERE",
              "original_condition": "(`tb_item`.`id` < 4)",
              "steps": [
                {
                  "transformation": "equality_propagation",
                  "resulting_condition": "(`tb_item`.`id` < 4)"
                },
                {
                  "transformation": "constant_propagation",
                  "resulting_condition": "(`tb_item`.`id` < 4)"
                },
                {
                  "transformation": "trivial_condition_removal",
                  "resulting_condition": "(`tb_item`.`id` < 4)"
                }
              ] /* steps */
            } /* condition_processing */
          },
          {
            "table_dependencies": [
              {
                "table": "`tb_item`",
                "row_may_be_null": false,
                "map_bit": 0,
                "depends_on_map_bits": [
                ] /* depends_on_map_bits */
              }
            ] /* table_dependencies */
          },
          {
            "ref_optimizer_key_uses": [
            ] /* ref_optimizer_key_uses */
          },
          {
            "rows_estimation": [
              {
                "table": "`tb_item`",
                "range_analysis": {
                  "table_scan": {
                    "rows": 9816098,
                    "cost": 2.04e6
                  } /* table_scan */,
                  "potential_range_indices": [
                    {
                      "index": "PRIMARY",
                      "usable": true,
                      "key_parts": [
                        "id"
                      ] /* key_parts */
                    }
                  ] /* potential_range_indices */,
                  "setup_range_conditions": [
                  ] /* setup_range_conditions */,
                  "group_index_range": {
                    "chosen": false,
                    "cause": "not_group_by_or_distinct"
                  } /* group_index_range */,
                  "analyzing_range_alternatives": {
                    "range_scan_alternatives": [
                      {
                        "index": "PRIMARY",
                        "ranges": [
                          "id < 4"
                        ] /* ranges */,
                        "index_dives_for_eq_ranges": true,
                        "rowid_ordered": true,
                        "using_mrr": false,
                        "index_only": false,
                        "rows": 3,
                        "cost": 1.6154,
                        "chosen": true
                      }
                    ] /* range_scan_alternatives */,
                    "analyzing_roworder_intersect": {
                      "usable": false,
                      "cause": "too_few_roworder_scans"
                    } /* analyzing_roworder_intersect */
                  } /* analyzing_range_alternatives */,
                  "chosen_range_access_summary": {
                    "range_access_plan": {
                      "type": "range_scan",
                      "index": "PRIMARY",
                      "rows": 3,
                      "ranges": [
                        "id < 4"
                      ] /* ranges */
                    } /* range_access_plan */,
                    "rows_for_plan": 3,
                    "cost_for_plan": 1.6154,
                    "chosen": true
                  } /* chosen_range_access_summary */
                } /* range_analysis */
              }
            ] /* rows_estimation */
          },
          {
            "considered_execution_plans": [
              {
                "plan_prefix": [
                ] /* plan_prefix */,
                "table": "`tb_item`",
                "best_access_path": {
                  "considered_access_paths": [
                    {
                      "access_type": "range",
                      "rows": 3,
                      "cost": 2.2154,
                      "chosen": true
                    }
                  ] /* considered_access_paths */
                } /* best_access_path */,
                "cost_for_plan": 2.2154,
                "rows_for_plan": 3,
                "chosen": true
              }
            ] /* considered_execution_plans */
          },
          {
            "attaching_conditions_to_tables": {
              "original_condition": "(`tb_item`.`id` < 4)",
              "attached_conditions_computation": [
              ] /* attached_conditions_computation */,
              "attached_conditions_summary": [
                {
                  "table": "`tb_item`",
                  "attached": "(`tb_item`.`id` < 4)"
                }
              ] /* attached_conditions_summary */
            } /* attaching_conditions_to_tables */
          },
          {
            "refine_plan": [
              {
                "table": "`tb_item`",
                "access_type": "range"
              }
            ] /* refine_plan */
          }
        ] /* steps */
      } /* join_optimization */
    },
    {
      "join_execution": {
        "select#": 1,
        "steps": [
        ] /* steps */
      } /* join_execution */
    }
  ] /* steps */
}