Increase insert speed

When the amount of data I collect reaches 8000 When all , When batch inserting data into a table with a unique index , Need to check for duplicates , So the insertion speed becomes slower and slower . Today, let's talk about how to improve INSERT Execution speed of .

Batch insert is faster than single insert

1. INSERT INTO table (field1,field2,field3) VALUES (‘a’,“b”,“c”), (‘a’,“b”,“c”),(‘a’,“b”,“c”);

2. INSERT INTO table (field1,field2,field3) VALUES (‘a’,“b”,“c”);INSERT INTO table (field1,field2,field3) VALUES (‘a’,“b”,“c”);INSERT INTO table (field1,field2,field3) VALUES (‘a’,“b”,“c”);

Use 1 Batch insert writing of , It can greatly improve the insertion speed .

Make rational use of the database buffer

It can be turned up bulk_insert_buffer_size This parameter . The default is 8M.
This parameter only applies to the use of MyISAM Storage engine , It is used to temporarily cache the write data when the batch insert data is cached .mysql This memory area will be used to cache the data of batch structure to help batch write data files .
For example, the buffer The size is set to 100M.
set global bulk_insert_buffer_size = 1024*1024*100;

use INSERT DELAYED Delay insertion

When a client uses INSERT DELAYED when , Will immediately get a confirmation from the server . And the rows are queued , When the table is not used by other threads , This line is inserted .
Use INSERT DELAYED Another important benefit of is , The insertions from many clients are lumped together , And written into a block . It's much faster than doing a lot of independent insertions .
We need to pay attention to ：

1. INSERT DELAYED Should only be used to specify the value list INSERT sentence . Server ignored for INSERT DELAYED…SELECT Of the statement DELAYED.
2. Server ignored for INSERT DELAYED…ON DUPLICATE UPDATE Of the statement DELAYED.
3. Because before the row is inserted , Statement immediately returns , So you cannot use LAST_INSERT_ID() To get AUTO_INCREMENT value .AUTO_INCREMENT Values may be generated by statements .
4. about SELECT sentence ,DELAYED Row invisible , Until these lines are actually inserted .
5. DELAYED Ignored in secondary replication server , because DELAYED The secondary server will not generate data different from the primary server .
6. Currently, the rows in the queue are only saved in memory , Until they are inserted into the table . It means , If you forcibly abort mysqld（ for example , Use kill -9） Or if mysqld The accident stopped , All rows that have not been written to the disk will be lost .