Sort out SQL performance optimization and collect classics!

Click on the top “ Terminal R & D department ”

 Set to “ Star standard ”, Master more database knowledge with you 

source ：juejin.im/post/59b11ba151882538cb1ecbd0

• Preface

  • 1 A brief introduction to the basic concepts

  • 2 Optimize on creation

  • 3 Optimize when querying

• Supplementary content

---

Preface

This paper mainly aims at relational database MySql. Key value class database can refer to ：

https://www.jianshu.com/p/098a870d83e4

First of all, simply comb Mysql Basic concepts of , And then divided into the creation time and the query time of the two stages of optimization .

1 A brief introduction to the basic concepts

1.1 Logical architecture

• first floor ： The client connects to the service , What will be done sql The command is transmitted

• The second floor ： The server parses and optimizes sql, Generate the final execution plan and execute

• The third level ： Storage engine , Responsible for data storage and extraction

1.2 lock

The database solves the concurrency scenario through the lock mechanism - Shared lock （ Read the lock ） And exclusive locks （ Write lock ）. Read lock is not blocked , Multiple clients can read the same resource at the same time . Writing lock is exclusive , And will block other read and write locks . Simply mention optimistic lock and pessimistic lock .

• Optimism lock , It is usually used in the scenario where data competition is not fierce , Read more and write less , Through version number and time stamp .

• Pessimistic locking , It's usually used in data competitive scenarios , Every operation locks the data .

To lock data, we need a certain locking strategy to cooperate with .

• Table locks , Lock the whole table , The cost is minimal , But it will intensify lock competition .

• Row lock , Lock row level , The most expensive , But it can support concurrency as much as possible .

however MySql The real implementation of the storage engine is not a simple row level lock , In general, multi version concurrency control is implemented （MVCC）.MVCC It's a variant of a row lock , In most cases, lock operation is avoided , Lower expenses .MVCC It is achieved by saving a snapshot of data at a certain point in time .

1.3 Business

Transactions guarantee a set of atomic operations , All or nothing , All or nothing . Once it fails , All operations before rollback .MySql Use auto submit , If you don't open a transaction explicitly , Then each query is treated as a transaction .

The isolation level controls changes in a transaction , Which are visible within and between transactions . Four common isolation levels ：

• Uncommitted read （Read UnCommitted）, Changes in transactions , Even if no commit is visible to other transactions . Transactions may read uncommitted data , Cause dirty reading .

• Submit to read （Read Committed）, When a transaction begins , You can only see the changes made by the submitted firm . Before the transaction is committed , The changes made are not visible to other transactions . It's also called non repeatable reading , Reading the same record multiple times for the same transaction may be different .

• Repeatable （RepeatTable Read）, The result is the same when the same record result is read multiple times in the same transaction .

• Serializable （Serializable）, The highest isolation level , Force transactions to execute serially .

1.4 Storage engine

InnoDB engine , above all , The most widely used storage engine . Designed to handle a lot of short-term business , With high performance and automatic crash recovery features .

MyISAM engine , Transaction and row level locks are not supported , Can't recover safely after a crash .

2 Optimize on creation

2.1 Schema And data type optimization

Integers

TinyInt,SmallInt,MediumInt,Int,BigInt Storage used 8,16,24,32,64 Bit storage space . Use Unsigned Negative numbers are not allowed , It can double the number of positive numbers .

The set of real Numbers

• Float,Double , Support approximate floating point operations .

• Decimal, Used to store exact decimals .

character string

• VarChar, Store variable length strings . need 1 or 2 Extra bytes record the length of the string .

• Char, Fixed length , Suitable for storing fixed length strings , Such as MD5 value .

• Blob,Text Designed to store large amounts of data . In the form of binary and character respectively .

Time type

• DateTime, Save a wide range of values , Occupy 8 Bytes .

• TimeStamp, recommend , And UNIX Same time stamp , Occupy 4 Bytes .

Optimization suggestions

• Try to use the corresponding data type . such as , Do not use string type to save time , Save with integer IP.

• Choose a smaller data type . It works TinyInt no need Int.

• Identity column （identifier column）, It is recommended to use integer , String type... Is not recommended , Take up more space , And the calculation speed is slower than the integer .

• Not recommended ORM The system automatically generates Schema, Usually with no focus on data types , Use big VarChar type , Index utilization is unreasonable and so on .

• Real scenes mix paradigms and anti paradigms . Redundant high query efficiency , Insert update is inefficient ; Low redundancy, high efficiency of insertion and update , Query efficiency is low .

• Create a completely separate summary \ Cache table , Generate data regularly , For user time-consuming operations . For summary operations requiring high accuracy , May adopt Historical results + The results of the latest record To achieve the purpose of quick query .

• Data migration , Shadow table can be used in the process of table upgrading , By modifying the table name of the original table , To save historical data , At the same time, it does not affect the purpose of using the new table .

2.2 Indexes

The index contains the values of one or more columns .MySql Only the leftmost prefix column of the index can be used efficiently . Advantages of index ：

• Reduce the amount of data scanned by queries

• Avoid sorting and zero time tables

• Will be random IO Change to order IO （ The order IO It's more efficient than random IO）

B-Tree

The most used index type . use B-Tree Data structure to store data （ Each leaf node contains a pointer to the next leaf node , Thus, it is convenient to traverse the leaf nodes ）.B-Tree The index applies to all key values , Key range , Key prefix lookup , Support sorting .

B-Tree Index limits ：

• If you don't start with the leftmost column of the index , Index cannot be used .

• You can't skip columns in an index . If you use the first and third column indexes , You can only use the first column index .

• If there is a range query in the query , All columns on the right cannot use index optimization queries .

Hash index

Only exactly match all columns of the index , The query is valid . The storage engine will compute a hash code for all index columns , Hash index stores all hash codes in the index , And save the pointer to each data row .

Hash index limit ：

• Can't be used to sort

• Partial matching is not supported

• Only equivalent queries such as =,IN（）, I won't support it < >

Optimization suggestions

• Note the scope and limitations of each index .

• If the column of index is part of expression or parameter of function , Failure .

• For very long strings , You can use prefix index , Select the appropriate prefix length according to the index selectivity .

• When using multi column indexes , Can pass AND and OR Grammatical connection .

• There's no need to repeat indexing , Such as （A,B） and （A） repeat .

• Index in where Condition query and group by Syntax queries are especially effective .

• Put the range query at the end of the condition query , Prevent the invalidation of right index caused by range query .

• It's better not to choose too long string for index , And the index column should not be null.

3 Optimize when querying

3.1 Three important indicators of query quality

• response time （ Service time , Queue time ）

• Scanning line

• Back to the line

3.2 Query optimization point

• Avoid querying unrelated Columns , If you use Select * Return all columns .

• Avoid querying unrelated rows

• Syncopation query . It will be a task with high pressure on the server , Break down into a longer period of time , And execute it several times . To delete 10000 pieces of data , Can be divided into 10 Execution times , Pause for a period of time after each execution , And go on with it . In the process, you can release server resources to other tasks .

• Decompose associated query . A query that associates multiple tables with a query , Decomposing multiple queries to single tables . Can reduce lock competition , The query itself is also more efficient . because MySql Connection and disconnection are lightweight operations , Will not be split into multiple times due to the query , Cause efficiency problems .

• Be careful count The operation can only be counted as null The column of , So count the total number of lines count（*）.

• group by Grouping by identity columns is efficient , The grouping result should not be used for the columns other than the grouping columns .

• Association query delay Association , You can first narrow the scope of each query according to the query criteria , Again .

• Limit Paging optimization . You can scan by index overlay , Then query other columns according to index column Association . Such as

SELECT
 id,
 NAME,
 age
WHERE
 student s1
INNER JOIN (
 SELECT
     id
 FROM
     student
 ORDER BY
     age
 LIMIT 50,5
) AS s2 ON s1.id = s2.id

• Union Query is de duplicated by default , If it's not the business that has to , It is recommended to use more efficient Union All

---

Supplementary content

From the great God - Xiaobao

1. The field type and table structure type in the condition are inconsistent ,mysql Will automatically add conversion function , Causes the index to fail as an argument in the function .

2.like No input in the previous part of the query , With % Cannot hit index at the beginning .

3. Add 2 individual 5.7 New features of version ：

generated column, This column in the database is calculated by other columns

CREATE TABLE triangle (sidea DOUBLE, sideb DOUBLE, area DOUBLE AS (sidea * sideb / 2));
insert into triangle(sidea, sideb) values(3, 4);
select * from triangle;

+-------+-------+------+
| sidea | sideb | area |
+-------+-------+------+
|   3      |   4      |  6     |
+-------+-------+------+

Support JSON Format data , And provide related built-in functions

CREATE TABLE json_test (name JSON);
INSERT INTO json_test VALUES('{"name1": "value1", "name2": "value2"}');
SELECT * FROM json_test WHERE JSON_CONTAINS(name, '$.name1');

come from JVM Experts - reach

Focus on explain Use in performance analysis

EXPLAIN SELECT settleId FROM Settle WHERE settleId = "3679"

• select_type, There are several values ：simple（ It means simple select, No, union And subquery ）,primary（ There are sub queries , The outermost select The query is primary）,union（union The second or subsequent select Inquire about , Independent of external query results ）,dependent union（union The second or subsequent select Inquire about , Rely on external query results ）

• type, There are several values ：system（ The watch has only one line （= The system tables ）, This is a const A special case of connection type ）,const（ Constant queries ）, ref( Non unique index access , Just a normal index ),eq_ref（ Use a unique index or component to query ）,all（ A full table query ）,index（ Query the whole table according to the index ）,range（ Range queries ）

• possible_keys: Index in the table that may help query

• key, Choose the index to use

• key_len, Index length used

• rows, Number of lines scanned , The bigger, the worse

• extra, There are several values ：Only index（ Information is retrieved from the index , Faster than a scan table ）,where used（ Use where Limit ）,Using filesort （ Maybe sort in memory or disk ）,Using temporary（ Use temporary tables when sorting query results ）

 reply  【idea Activate 】 You can get idea How to activate 
 reply  【Java】 obtain java Relevant video tutorials and materials 
 reply  【SpringCloud】 obtain SpringCloud Many relevant learning materials 
 reply  【python】 Get the full set 0 Basics Python Knowledge Manual 
 reply  【2020】 obtain 2020java Related interview questions tutorial 
 reply  【 Add group 】 You can join the technical exchange group related to the terminal R & D department 
 Read more 
 use  Spring  Of  BeanUtils  front , I suggest you understand these pits first ！

lazy-mock , A lazy tool for generating backend simulation data 

 In Huawei Hongmeng  OS  Try some fresh food , My first one “hello world”, take off ！

 The byte is bouncing ：i++  Is it thread safe ？

 One  SQL  Accidents caused by , Colleagues are fired directly ！！

 Too much ！ Check Alibaba cloud  ECS  Of  CPU  Incredibly reach 100%

 a vue Write powerful swagger-ui, A little show （ Open source address attached ）


 Believe in yourself , Nothing is impossible , Only unexpected, not only technology is obtained here ！



 If you like, just give me “ Looking at ”