Rethinking about MySQL query optimization

About mysql Rethinking of query optimization

I wrote an article about mysql Optimization thinking article ,mysql Ten experiences of sentence optimization , Now I want to come for a few years , I have been in contact with several optimization problems at ordinary times , It's time to continue writing a new article .

About index hits

Many times for short queries (where There are only a few sentences , There is no table Association ), Just hit an index , The speed will be very fast , At the same time, even if the small table does not hit , Basically, it doesn't matter ( The quantity set problem will be mentioned later )
But for the vast majority of short queries used, we know that they will basically hit the index ( Unless the table has no index ). Therefore, the real problem is often multi table joint query , In many cases , complex on Index hits under statements are often random ( It can't be said to follow fate , But for non dba Development of , The optimization mechanism of index is black box , Only try to guess and write by experience ), Many you think will hit , It often doesn't hit , Use at this time explain Often the advantage . But I have also encountered hit in the test environment , In the case of a miss in the formal environment , Guess is under different quantities , The optimization mechanism is different .

Joint index

Once the union index hits , The speed will improve very fast , But the hit rule of the joint index is to hit left , So many times people will limit the way they write ( At least in my current 5.7 It's like this , I don't know what will happen next mysql Will it optimize )

Use of slow queries

Open slow query and cooperate with various logs to quickly find the query statements with problems . Especially for those slow queries that often appear .

However, there should also be a concept of cost in development . For example, many of our current slow queries are generated by generating query logs in the early morning . The emergence of these slow queries is actually within our estimated range , There is no need to invest development costs to optimize . For example, I mentioned this article before mysql The associated index hit .md It often happens , It is necessary to optimize the query specifically . Although it took several hours to query and optimize , But it's worth it .

Order of magnitude

Code development , Order of magnitude is a commonly used concept , For example, backstage , Every page has 20 strip , I use short queries for every one , Although there are 20*n Short query for , But there is no obvious difference in speed compared with some associated queries that do not hit the index , It will be faster to write . Problems are also easier to find .

Another example is the optimization I mentioned above , Because of the size table driven problem , Association query , Driven by size table , Once the index is not hit , Generate millions of full table queries , The speed will become extremely touching . This is also the reason for the size table driving problem often mentioned in many articles .

Weigh development time , Frequency of occurrence can also be regarded as an order of magnitude concept , Put a bunch of long queries into the cache , After that, avoiding re query is also a common optimization method .

Syncopation query

Sometimes we need to divide and conquer a big query , Every query is exactly the same , Only a small part is returned at a time .
For example, we need to delete messages of a certain month , Disposable delete There is a lot of pressure on the server , We can delete only 10000 strip , Loop processing , Greatly reduce the impact on the server , Colleagues reduce the holding time of locks when deleting .

At the same time, avoid accessing too much data at one time , It can also effectively avoid orders of magnitude , As I mentioned before, try to avoid using *, For multi table queries , No, limit Time is especially important , Under large quantity set , Send only 1/4 The data of , It can also greatly save time .

Processing in a programming language

I mentioned in my previous article , In many cases , Extract the data and process it in the programming language , It's also an advantage , Before helping interns optimize code, there was a problem . The logic of interns is to put data processing and inspection into mysql Inside , It takes a lot of time and energy to study how to hit the index . My experience is , Use a short query , Process a table , use yii Inside index sentence , Extract the previous data and clean out the useless data , Then put the remaining dozen useful data to query the database , In this way, the speed is increased to 20 times .
Waste a little cpu And memory resources . It greatly reduces the pressure and development cost of the database .