MySQL index failure and solution practice

1. Preface

The following tests use MariaDB, Version is 10.0.36,InnoDb Version is 5.6.39-83.1.

2. Description of database table for test

The following database table for testing is test_table_log、test_table_log2, The fields of the two tables are the same , Different indexes , As shown below ：

CREATE TABLE IF NOT EXISTS test_table_log
(
   id             varchar(32) NOT NULL COMMENT 'id',
   type           varchar(20) NOT NULL COMMENT 'type',
   cust_info      varchar(20) NOT NULL COMMENT 'cust_info',
   status         varchar(2) NOT NULL COMMENT 'status',
   other1         varchar(20) NOT NULL COMMENT 'other1',
   other2         varchar(20) NOT NULL COMMENT 'other2',
   create_time    DATETIME(3) NOT NULL COMMENT 'create_time',
   update_time    DATETIME(3) NOT NULL COMMENT 'update_time',
   PRIMARY KEY(id)
)
COMMENT = ' Test flow chart ';

CREATE TABLE IF NOT EXISTS test_table_log2 like test_table_log;

alter table test_table_log add INDEX idx_ttl_type(type);
alter table test_table_log add INDEX idx_ttl_cust(cust_info);
alter table test_table_log add INDEX idx_ttl_status(status);
alter table test_table_log add INDEX idx_ttl_time(create_time);

alter table test_table_log2 add INDEX idx_ttl2_all(cust_info,create_time,type,status);
alter table test_table_log2 add INDEX idx_ttl2_time(create_time);

The number of rows in the above two tables is 500 Line ten thousand , Same data ,type The scope is [‘1’,‘10’],cust_info The scope is [‘1000’,‘2000’],status The scope is [‘1’,‘3’],other1 Fixed for ’1’,other2 The scope is [‘1000’,‘2000’],create_time、update_time It can be regarded as the time of insertion .

3. Implementation of index validation

In the following cases , When an index is specified in a query statement , The index can work .

3.1. Single index

The scope of the following query criteria should be small , The index takes effect .

3.1.1. The query condition uses =

select * from test_table_log 
where cust_info='1881'; 

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: ref
possible_keys: idx_ttl_cust
          key: idx_ttl_cust
      key_len: 22
          ref: const
         rows: 10556
        Extra: Using index condition

The query operation takes the following time ：

Query_time: 2.507578  Lock_time: 0.001224  Rows_sent: 5222  Rows_examined: 5222

The index reading times of query statements are as follows ：

Handler_read_key         | 1
Handler_read_next        | 5222

InnoDB The number of index pages and records in the buffer pool is as follows ：

| TABLE_NAME                | INDEX_NAME   | page_num | record_num |
+---------------------------+--------------+----------+------------+
| `testdb`.`test_table_log` | PRIMARY      | 4798     | 955437     |
| `testdb`.`test_table_log` | idx_ttl_cust |   15     |   5785     |

3.1.2. Range queries

select * from test_table_log 
where cust_info>='1881' and cust_info<='1882';

select * from test_table_log 
where cust_info between '1881' and '1882';

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: range
possible_keys: idx_ttl_cust
          key: idx_ttl_cust
      key_len: 22
          ref: NULL
         rows: 21096
        Extra: Using index condition

3.1.3. The query condition uses or

For the same index field

select * from test_table_log 
where cust_info='1881' or cust_info='1882';

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: range
possible_keys: idx_ttl_cust
          key: idx_ttl_cust
      key_len: 22
          ref: NULL
         rows: 20480
        Extra: Using index condition; Using where

select * from test_table_log 
where create_time <='2020-03-10 12:20:39.000' and (cust_info='1881' or type='1');

3.1.4. The query condition uses in

select * from test_table_log 
where cust_info in ('1881','1882');

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: range
possible_keys: idx_ttl_cust
          key: idx_ttl_cust
      key_len: 22
          ref: NULL
         rows: 20480
        Extra: Using index condition

3.1.5. The query condition uses is null/is not null

When satisfied is null/is not null When the amount of index field data of condition is small , You can use index .

For example, the total number of rows created is 1000 Table of .

An index field that is allowed to be empty is about 50% The record of is not empty , about 50% The record of is empty ,is null Can use index ,type by ref; When the query condition is equal to a constant or is null when , Can use index ,type by ref_or_null.

An index field that is allowed to be empty is about 10% The record of is not empty , about 90% The record of is empty ,is null Can use index ,type by ref;is not null Index can also be used ,type by range.

3.2. Joint index

For the case that the union index is used and takes effect , The preceding fields must be specified in the query conditions , And the query scope should be small . The details are omitted .

4. Ignore the execution of the index

The following is the execution when ignoring the index , Can use full scan .

select * from test_table_log ignore index (idx_ttl_cust) 
where cust_info='1881'; 

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 4760085
        Extra: Using where

The query operation takes the following time ：

Query_time: 3.470909  Lock_time: 0.001395  Rows_sent: 5222  Rows_examined: 5000000

The index reading times of query statements are as follows ：

Handler_read_rnd_next    | 5000001

InnoDB The number of index pages and records in the buffer pool is as follows ：

| TABLE_NAME                | INDEX_NAME | page_num | record_num |
+---------------------------+------------+----------+------------+
| `testdb`.`test_table_log` | PRIMARY    | 7075     | 1389569    |

5. Index failure and solutions

5.1. Different codes

Reference resources https://dev.mysql.com/doc/refman/5.6/en/mysql-indexes.html , When comparing non binary character strings , The corresponding columns should use the same character set . For example, use utf8 Encoded columns and use latin1 Indexes are not used when comparing encoded columns .

5.2. Implicit conversion

select * from test_table_log 
where cust_info=1881; 

The fields used in the above query cust_info Type is string , But the query condition is int, Implicit conversion will be performed during query , Index not available .

The execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust.

The time-consuming of query operation is close to the execution of ignoring index above .

The number of times the query statement reads the index is the same as the above execution of ignoring the index .

InnoDB The execution of index pages in the buffer pool is the same as that of ignoring indexes above .

To prevent implicit conversion , Query parameters need to use the correct type .

5.3. Function operation

select * from test_table_log 
where concat(cust_info,'0')='18810';

select * from test_table_log where time_to_sec(timediff(create_time,'2020-03-10 12:20:40.000'))<3;

The above query performs function operations on the fields corresponding to the single column index , Index not available .

The execution plan is displayed as a full table scan ,possible_keys by NULL.

If you need to use function operation , It should be added to the query criteria , You can use index , As shown below ：

select * from test_table_log 
where cust_info=left('18810',4);

select * from test_table_log where create_time<date_add('2020-03-10 12:20:40.000', interval 3 second);

5.4. like Start with a wildcard

Does not satisfy the leftmost matching principle

select * from test_table_log 
where cust_info like '%1881'; 

The execution plan is displayed as a full table scan ,possible_keys by NULL.

In order to use like Can use index ,like The corresponding query criteria cannot be ’%' start .

5.5. Query different fields using or

When using or When querying two different fields , The index will fail （ close index_merge）.

5.5.1. Single column index and non index fields or

select * from test_table_log 
where cust_info='1881' or other1='1';

When querying, use the fields corresponding to a single column index and non index fields or In operation , No index .

The execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust.

5.5.2. Between single column index fields or

When querying, use the corresponding fields of two single column indexes or In operation , Whether to use index and index_merge Optimizer switch related . The query statement is as follows .

select * from test_table_log 
where cust_info='1881' or create_time<='2020-03-10 12:20:39.000';

5.5.2.1. Turn on index_merge

index_merge The optimizer is on by default .

When open index_merge when , The above execution plan shows , The connection type is index merge , Used idx_ttl_cust And idx_ttl_time Indexes .

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	test_table_log	index_merge	idx_ttl_cust,idx_ttl_time	idx_ttl_cust,idx_ttl_time	22,7	NULL	12488	Using sort_union(idx_ttl_cust,idx_ttl_time); Using where

The query operation takes the following time ：

Query_time: 2.520033  Lock_time: 0.003090  Rows_sent: 7152  Rows_examined: 7152

5.5.2.2. close index_merge

You can close index_merge, or index_merge_sort_union Optimizer switch .

set optimizer_switch = 'index_merge=off';
set optimizer_switch = 'index_merge_sort_union=off';

When off index_merge when , The above execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust,idx_ttl_time.

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	test_table_log	ALL	idx_ttl_cust,idx_ttl_time	NULL	NULL	NULL	4760085	Using where

The query operation takes the following time ：

Query_time: 3.925060  Lock_time: 0.001728  Rows_sent: 7152  Rows_examined: 5000000

5.5.3. Use or Lead to index failure optimization

Use two fields with single column indexes in the query or when , Use union all, It can solve the problem of index failure , It can replace the method of starting index merging .

5.5.3.1. Query data

adopt union all When querying data , The sample statements are as follows ：

select * from test_table_log 
where cust_info='1881'
union all
select * from test_table_log 
where create_time<='2020-03-10 12:20:39.000';

5.5.3.2. Number of query lines

adopt union all When querying data , The sample statements are as follows ：

select count(*) from test_table_log 
where cust_info='1881'
union all
select count(*) from test_table_log 
where create_time<='2020-03-10 12:20:39.000';

The above query results are multiple lines , have access to sum Get the sum , As shown below ：

select sum(a.c) from 
(
select count(*) as c from test_table_log 
where cust_info='1881'
union all
select count(*) as c from test_table_log 
where create_time<='2020-03-10 12:20:39.000'
)as a;

5.5.4. Union index internal fields or

When used between different fields within a federated index or when , Whether it's on or not index_merge, Indexes will be invalidated . For example (a,b) Add a federated index to the column , The query conditions are “a = xxx or b = xxx” when , The index will fail .

5.6. is null/is not null Too much scope

When satisfied is null/is not null When there is too much index field data in , No index .

select * from test_table_log 
where cust_info is not null;

The execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust.

5.7. Use != or <>

Use != or <> when , The index will fail . The sample statements are as follows ：

select * from test_table_log 
where cust_info != '1881';

select * from test_table_log 
where cust_info <> '1881';

The execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust.

5.8. The query condition uses not in

The query condition uses not in when , The index will fail . The sample statements are as follows ：

select * from test_table_log 
where cust_info not in ('1881');

The execution plan is displayed as a full table scan ,possible_keys by idx_ttl_cust.

5.9. The query condition uses not like

The query condition uses not like when , The index will fail . The sample statements are as follows ：

select * from test_table_log 
where cust_info not like '1881%';

The execution plan is displayed as a full table scan ,possible_keys by NULL.

5.10. The range of query criteria is too large

When the data range corresponding to the query condition is too large , The index will fail . The sample statements are as follows ：

select max(other2) from test_table_log 
where cust_info<'1919';

select max(other2) from test_table_log 
where cust_info>'1919';

above SQL Statements do not use indexes idx_ttl_cust.

  select_type: SIMPLE
        table: test_table_log
         type: ALL
possible_keys: idx_ttl_cust
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 4760085
        Extra: Using where

The following is for ignoring indexes （ With the above query criteria range is too large, resulting in index failure SQL The effect of the statement is the same ）, And the forced use of indexes .

You can see that when the data range corresponding to the query criteria is too large , Sometimes using a full table scan takes less time than querying through a secondary index .

• Ignore the index

select max(other2) from test_table_log ignore index (idx_ttl_cust)
where cust_info>'1919';

• Force index

select max(other2) from test_table_log force index (idx_ttl_cust)
where cust_info>'1919';

5.10.1. Execution plan comparison

• Ignore the index

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	test_table_log	ALL	NULL	NULL	NULL	NULL	4760085	Using where

• Force index

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	test_table_log	range	idx_ttl_cust	idx_ttl_cust	22	NULL	831036	Using index condition

5.10.2. Comparison of query operation time

• Ignore the index

Query_time: 2.683089  Lock_time: 0.003528  Rows_sent: 1  Rows_examined: 5000000

• Force index

Query_time: 86.099621  Lock_time: 0.001233  Rows_sent: 1  Rows_examined: 404372

5.10.3. Comparison of index reading times of query statements

• Ignore the index

Handler_read_rnd_next    | 5000001

• Force index

Handler_read_key         | 1
Handler_read_next        | 404372

5.11. The union index does not use the previous field

When using the corresponding field of the joint index to query , If the previous fields are not used for query , The index will be invalid . For example, there is an index (a,b,c) when , Use a、a,b、a,b,c Index can be used when querying , Use b、b,c、c The index will be invalid when querying .

Following SQL Statement query uses create_time,type,status Field , There is a federated index (cust_info,create_time,type,status), Because the query criteria are not used cust_info Field , Therefore, the federated index will fail .

select * from test_table_log2 
where create_time>'2020/3/8 11:57:27' and type='1' and status='1';

The execution plan is as follows ,（create_time Field has a single column index , But the data range corresponding to the query criteria is too large , Therefore, indexes will not be used ）：

  select_type: SIMPLE
        table: test_table_log2
         type: ALL
possible_keys: idx_ttl2_time
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 4727385
        Extra: Using where

5.12. in or or The amount of data is too small when querying

The amount of data in the database table is too small （ For example, less than or equal to 6 That's ok ） when , Use... For the same index field or or in When querying by operation , No index .

Reference resources https://dev.mysql.com/doc/refman/5.6/en/table-scan-avoidance.html, For tables with small amount of data , A full table scan is usually appropriate , And the impact on performance can be ignored .

As below SQL The statement will not use the index when executing .

select * from test_table_log where cust_info in ('1881','1882');

select * from test_table_log where cust_info='1881' or cust_info='1882';

The execution plan is as follows ：

  select_type: SIMPLE
        table: test_table_log
         type: ALL
possible_keys: idx_ttl_cust
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 6
        Extra: Using where