Quickly build MySQL million level test data

explain

The problems and solutions described in this paper are also applicable to Tencent cloud Cloud database MySQL（TencentDB for MySQL,CDB）.

background

During the verification of query and other operations , We often need to build a large amount of basic data in the offline environment for us to test , Simulate the real environment on the line .

Build a data

Here we quickly build a test data , It is used to simulate the actual production in the middle order 100 A data sheet of Wan .

1. Create test libraries and basic tables

MySQL [(none)]> CREATE DATABASE dts_demo;
Query OK, 1 row affected (0.00 sec)

MySQL [(none)]> USE dts_demo;
Database changed
MySQL [dts_demo]> CREATE TABLE `user_info` (
    ->   `id` int(11) NOT NULL AUTO_INCREMENT,
    ->   `c_user_id` varchar(36) NOT NULL DEFAULT '',
    ->   `c_name` varchar(22) NOT NULL DEFAULT '',
    ->   `c_province_id` int(11) NOT NULL,
    ->   `c_city_id` int(11) NOT NULL,
    ->   `create_time` datetime NOT NULL,
    ->   PRIMARY KEY (`id`),
    ->   KEY `idx_user_id` (`c_user_id`)
    -> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.01 sec)

2. Create a memory table

utilize MySQL Memory table insertion speed is fast , First, we use functions and stored procedures to generate data in the memory table , Then insert it from the memory table into the normal table .

MySQL [dts_demo]> CREATE TABLE `user_memory` (
    ->   `id` int(11) NOT NULL AUTO_INCREMENT,
    ->   `c_user_id` varchar(36) NOT NULL DEFAULT '',
    ->   `c_name` varchar(22) NOT NULL DEFAULT '',
    ->   `c_province_id` int(11) NOT NULL,
    ->   `c_city_id` int(11) NOT NULL,
    ->   `create_time` datetime NOT NULL,
    ->   PRIMARY KEY (`id`),
    ->   KEY `idx_user_id` (`c_user_id`)
    -> ) ENGINE=MEMORY;
Query OK, 0 rows affected (0.00 sec)

3. Create a function

Create a function of random string and random time

MySQL [dts_demo]> delimiter $$
MySQL [dts_demo]> 
MySQL [dts_demo]> CREATE DEFINER=`root`@`%` FUNCTION `randStr`(n INT) RETURNS varchar(255) CHARSET utf8mb4
    ->     DETERMINISTIC
    -> BEGIN
    ->     DECLARE chars_str varchar(100) DEFAULT 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
    ->     DECLARE return_str varchar(255) DEFAULT '' ;
    ->     DECLARE i INT DEFAULT 0;
    ->     WHILE i < n DO
    ->         SET return_str = concat(return_str, substring(chars_str, FLOOR(1 + RAND() * 62), 1));
    ->         SET i = i + 1;
    ->     END WHILE;
    ->     RETURN return_str;
    -> END$$
Query OK, 0 rows affected (0.00 sec)

MySQL [dts_demo]> CREATE DEFINER=`root`@`%` FUNCTION `randDataTime`(sd DATETIME,ed DATETIME) RETURNS datetime
    ->     DETERMINISTIC
    -> BEGIN
    ->     DECLARE sub INT DEFAULT 0;
    ->     DECLARE ret DATETIME;
    ->     SET sub = ABS(UNIX_TIMESTAMP(ed)-UNIX_TIMESTAMP(sd));
    ->     SET ret = DATE_ADD(sd,INTERVAL FLOOR(1+RAND()*(sub-1)) SECOND);
    ->     RETURN ret;
    -> END $$
Query OK, 0 rows affected (0.00 sec)

4. Create stored procedure

Create stored procedures to insert data

MySQL [dts_demo]> CREATE DEFINER=`root`@`%` PROCEDURE `add_user_memory`(IN n int)
    -> BEGIN
    ->     DECLARE i INT DEFAULT 1;
    ->     WHILE (i <= n) DO
    ->         INSERT INTO user_memory (c_user_id, c_name, c_province_id,c_city_id, create_time) VALUES (uuid(), randStr(20), FLOOR(RAND() * 1000), FLOOR(RAND() * 100), NOW());
    ->         SET i = i + 1;
    ->     END WHILE;
    -> END
    -> $$
Query OK, 0 rows affected (0.00 sec)
MySQL [dts_demo]> delimiter ;

5. Calling stored procedure

Call the stored procedure to write the test data to the memory table

MySQL [dts_demo]> CALL add_user_memory(1000000);
Query OK, 1 row affected (1 min 50.74 sec)

production 100 Million test data 50 second , It is more efficient .

6. Write formal table

Insert normal table from memory table

MySQL [dts_demo]> INSERT INTO user_info SELECT * FROM user_memory;
Query OK, 1000000 rows affected (7.02 sec)
Records: 1000000  Duplicates: 0  Warnings: 0

MySQL [dts_demo]> DROP TABLE user_memory;
Query OK, 0 rows affected (0.00 sec)

7. Disrupt creation time

Update the creation time field to make the creation time of the inserted data more random

MySQL [dts_demo]> UPDATE user_info SET create_time=date_add(create_time, interval FLOOR(1 + (RAND() * 7)) year);
Query OK, 1000000 rows affected (2.94 sec)
Rows matched: 1000000  Changed: 1000000  Warnings: 0

MySQL [dts_demo]> select * from user_info limit 20;
+----+--------------------------------------+----------------------+---------------+-----------+---------------------+
| id | c_user_id                            | c_name               | c_province_id | c_city_id | create_time         |
+----+--------------------------------------+----------------------+---------------+-----------+---------------------+
|  1 | 1afd2630-88bc-11eb-9c30-0c42a125994e | oxlXASuDAQhIAEmDVAZ4 |             8 |        33 | 2022-03-19 22:05:05 |
|  2 | 1afd300e-88bc-11eb-9c30-0c42a125994e | Nj27hTrqAwIQUPiO0qXo |           727 |        95 | 2028-03-19 22:05:05 |
|  3 | 1afd4041-88bc-11eb-9c30-0c42a125994e | J9rzo41MCC2dM5Whp4Zy |           482 |        22 | 2026-03-19 22:05:05 |
|  4 | 1afd4562-88bc-11eb-9c30-0c42a125994e | RX3eSuFHkqXmNJ8hSoas |           517 |        67 | 2023-03-19 22:05:05 |
|  5 | 1afd4a49-88bc-11eb-9c30-0c42a125994e | YcVRm6gPdssI6cxUMZs9 |            54 |        31 | 2023-03-19 22:05:05 |
|  6 | 1afd4ebd-88bc-11eb-9c30-0c42a125994e | ydfrgRm1VlPX8FLFSeo5 |           968 |         3 | 2027-03-19 22:05:05 |
|  7 | 1afd530c-88bc-11eb-9c30-0c42a125994e | rsMpwgyPk0TiBXO2AFr3 |           585 |        25 | 2027-03-19 22:05:05 |
|  8 | 1afd574a-88bc-11eb-9c30-0c42a125994e | H5aqu0qT4xgB06i1341J |           293 |        73 | 2027-03-19 22:05:05 |
|  9 | 1afd5cf9-88bc-11eb-9c30-0c42a125994e | Y10PZgc4AzTDjxyY5ke0 |            31 |        60 | 2025-03-19 22:05:05 |
| 10 | 1afd61a8-88bc-11eb-9c30-0c42a125994e | 761DXGqU7GUjHpKns2E0 |           732 |        12 | 2022-03-19 22:05:05 |
| 11 | 1afd662c-88bc-11eb-9c30-0c42a125994e | AVIBJG21NLi00PX8HS7O |           384 |        97 | 2022-03-19 22:05:05 |
| 12 | 1afd6ace-88bc-11eb-9c30-0c42a125994e | RK0E38ooDO0r1CSn6dz6 |           474 |        53 | 2022-03-19 22:05:05 |
| 13 | 1afd6f01-88bc-11eb-9c30-0c42a125994e | pNCyKUaVYVyQqowgB3kl |           370 |        31 | 2028-03-19 22:05:05 |
| 14 | 1afd7332-88bc-11eb-9c30-0c42a125994e | CvwX2bCq4VhshQeuy9Yf |           960 |        55 | 2024-03-19 22:05:05 |
| 15 | 1afd775f-88bc-11eb-9c30-0c42a125994e | 3YzKT2oEXGmAIDRdo9on |           383 |        26 | 2024-03-19 22:05:05 |
| 16 | 1afd7bcf-88bc-11eb-9c30-0c42a125994e | j8zjGigivtHUhwDq2OK9 |           172 |        90 | 2025-03-19 22:05:05 |
| 17 | 1afd800c-88bc-11eb-9c30-0c42a125994e | 9pqJfSuEE8AlMKdHHeTD |           130 |        24 | 2025-03-19 22:05:05 |
| 18 | 1afd842c-88bc-11eb-9c30-0c42a125994e | 0DZUqdFwtEGifda3AA4p |           480 |        67 | 2028-03-19 22:05:05 |
| 19 | 1afd886b-88bc-11eb-9c30-0c42a125994e | 6SRyZ7v0mCP981zBaSIL |           374 |         5 | 2022-03-19 22:05:05 |
| 20 | 1afd8c9f-88bc-11eb-9c30-0c42a125994e | jKFUparzjJAyRrv4DMST |           530 |        43 | 2024-03-19 22:05:05 |
+----+--------------------------------------+----------------------+---------------+-----------+---------------------+
20 rows in set (0.00 sec)

thus ,Mysql The test table has been simulated successfully .