MySQL limit usage and large paging problem solving

limit grammar

limit The syntax supports two parameters ,offset and limit, The former represents the offset , The latter means taking the former limit Data .

for example ：

##  Return to the qualified former 10 statement  
select * from user limit 10

##  Return the qualified number 11-20 Data 
select * from user limit 10,20

It can also be seen from above ,limit n Equivalent to limit 0,n.

Performance analysis

In actual use, we will find , Some pages after paging , Loading will slow down , in other words :

select * from user limit 1000000,10

Statement execution is slow . So let's test it first .

The first is offset In smaller cases, take 100 Data .( The total amount of data is 200 about ). Then gradually increase offset.

select * from user limit 0,100 --------- Time consuming 0.03s
select * from user limit 10000,100 --------- Time consuming 0.05s
select * from user limit 100000,100 --------- Time consuming 0.13s
select * from user limit 500000,100 --------- Time consuming 0.23s
select * from user limit 1000000,100 --------- Time consuming 0.50s
select * from user limit 1800000,100 --------- Time consuming 0.98s

You can see that with offset The increase of , Performance is getting worse .

Why is that ? because limit 10000,10 The grammar of is actually mysql Before finding 10010 Data , Then discard the front 10000 That's ok , This step is actually wasted .

Optimize

1. use id Optimize

First find the largest page number of the last page ID, And then use it id To query , Be similar to

select * from user where id>1000000 limit 100.

It's very fast , Because the primary key is indexed , But there's a drawback to that , Namely ID It has to be continuous Of , also Query cannot have where sentence ,
because where Statements cause filtered data .

2. Optimizing with overlay index

mysql When the query completely hits the index , Called overlay index , It's very fast , Because the query only needs to look up on the index , Then you can go straight back to , Instead of going back to the data table to get the data . So we can look up the index first ID, And then according to Id Get data .

select * from (select id from job limit 1000000,100) a left join job b on a.id = b.id;

Time consuming 0.2 second .

summary

use mysql Paging a lot of data is really difficult , But there are also some ways to optimize , It needs to be tested more in combination with business scenarios .
When the user turns to 10000 Page time , Why don't we just return to empty , So boring …