MySql tens of millions of paging optimization, fast insertion method of tens of millions of data

In order to facilitate testing, create a new table and insert 20 million pieces of test data

Mysql version: 5.7.34

---

1. Quickly insert millions of data

--Create a MyISAM schema table to facilitate batch running of data (simulate the actual table with multiple fields)CREATE TABLE `my_tables` (`id` bigint(32) NOT NULL AUTO_INCREMENT,`name` varchar(32) DEFAULT NULL,`age` int(32) DEFAULT NULL,`time` varchar(32) DEFAULT NULL,`pwd` varchar(32) DEFAULT NULL,`test` varchar(255) DEFAULT NULL,`test2` varchar(255) DEFAULT NULL,`test3` varchar(255) DEFAULT NULL,`test4` varchar(255) DEFAULT NULL,`test5` varchar(255) DEFAULT NULL,`test6` varchar(255) DEFAULT NULL,`test7` varchar(255) DEFAULT NULL,`test8` varchar(255) DEFAULT NULL,`test9` varchar(255) DEFAULT NULL,`test10` varchar(255) DEFAULT NULL,PRIMARY KEY (`id`)) ENGINE=MyISAM AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;--Create a stored procedure (insert two million pieces of data, about 45 seconds, the more data is executed a few times)DROP PROCEDURE IF EXISTS my_insert;CREATE PROCEDURE my_insert()BEGINDECLARE n int DEFAULT 1;loopname:LOOPINSERT INTO `my_tables`(`name`,`age`,`time`,`pwd`)VALUES ('Test Name', 18, '2022-07-27', '369');SET n=n+1;IF n=2000000 THENLEAVE loopname;END IF;END LOOP loopname;END;--Execute the stored procedureCALL my_insert();

2. Paging statement

The writing method of select * in the following sql is just for testing and display Do not actually use it like this (write which fields should be checked)!

//Number of queriesSELECT count(1) FROM my_tables;25000021 --- 0.03 seconds (count query is really fast)// only query the indexSELECT id FROM my_tables limit 20000000,20;2.965 seconds//Ordinary paging querySELECT * FROM my_tables limit 20000000,20;4.441 seconds//Optimization 1: Subquery methodSELECT * FROM my_tables where id>=(select id from my_tables limit 20000000,1)limit 20;2.981 seconds//Optimization 2: join query method (compared to sub-query query speed is about the same, the speed is increased by about 30%)SELECT * FROM my_tables a JOIN(select id from my_tables limit 20000000,20)b on a.id=b.id;2.953 seconds//Optimization 3: Sort method according to primary key or unique index (use index to quickly locate part of data, avoid full table scan, and increase the speed by 99%)SELECT * FROM my_tables where id>=20000000 ORDER BY id ASC LIMIT 0,20;0.038 seconds

3. Actual usage:

pageNum: page numberpageSize: How many items are displayed on one pageSELECT * FROM table name WHERE id>=(pageNum*pageSize) ORDER BY id ASC LIMIT pageNum,pageSize;

4. Error example:

//When connecting table query or sub-table query, the limit should be placed in it, otherwise the query time and data volume will multiply the number of data in the two tables.mistake:SELECT * FROM a table a JOIN(select id from a table) b on a.id=b.id limit 0,20;correct:SELECT * FROM a table a JOIN(select id from a table limit 0,20) b on a.id=b.id;mistake:select * from (select * from a table a left join b table b on a.id = b.id ) tt where ... limit 0,20correct:select * from (select * from a table a left join b table b on a.id = b.id limit 0,13) tt where ...

---

The optimization ends here. The above sql is written for testing without adding where conditions, etc. You need to rewrite it according to your actual needs

If the article is wrong or in doubt, please leave a message to discuss~