I've taken it. MySQL table 500W rows, but someone doesn't partition it?

Preface

Table partitioning , According to certain rules , Decompose a table in a database into smaller ones , Easy to manage parts , This paper makes a detailed description

 1. Partition table

 1.1 What is table partitioning

We can go through show variables like ‘%datadir%’;

Command to view the default path where the data file is stored , A database is a folder , In a library .

As long as the amount of data in a table is too large , It will lead to *.ibd File is too large. , The search of data will become very slow .

In general, it is recommended that a single day exceed 10 Line ten thousand , Suggest zoning ,1 Billion data, about 1G

————————————————

MySQL from 5.1 Started adding support for partitions ,

The process of partitioning is to decompose a table or index into smaller ones 、 More manageable parts .

For developers , Basically, the table as like as two peas is not identical. , Just on physical storage ,

Originally, the table had only one data file , Now it has become multiple , Each partition is a separate object , You can handle it on your own , It can also be processed as part of a larger object .

Common storage engines such as InnoDB、MyISAM、NDB And so on .
But not all storage engines support , Such as CSV、FEDORATED、MERGE Partition is not supported ,
So before using this partition function , You should be aware of the partition support of the selected storage engine .

 1.2 Why do I need a table partition

1. You can make a single table store more data .

2. Partition table data is easier to maintain , You can delete a large amount of data in batch by clearing the entire partition ,

You can also add new partitions to support the newly inserted data . in addition , You can also optimize a separate partition 、 Check 、 Repair and other operations .

3. Some queries can only fall on a few partitions according to the query criteria , The query speed will be fast .

4. Partition table data can also be distributed on different physical devices , So as to make efficient use of multiple hardware devices .

5. Partition tables can be used to avoid some special bottlenecks ,

for example InnoDB Exclusive access to a single index 、ext3 File system inode Lock competition .

You can back up and restore a single partition .

 1.3 Disadvantages of partitioned tables

The main disadvantage of table partitioning

1. A watch can only have at most 1024 Zones .
2. If the partition field has a primary key or a unique index column , Then all primary key columns and unique index columns must be included .
3. Partition tables cannot use foreign key constraints .
4.NULL Value invalidates partition filtering .
5. All partitions must use the same storage engine .

The main advantages of table partitioning

1、 You can allow more data to be stored in a table , Break through disk and file system limitations .
2、 It's easy to delete expired historical data from the table , Just remove the corresponding partition .
3、 For some queries or modification statements , It can automatically narrow the data range to one or several partitions , Optimize statement execution efficiency .

 2. Type of partition table

 2.1 RANGE Partition

Range table partition , Determine the data contained in each partition according to a certain range value , As used above is range Table partitioning ;

grammar ：partition by range(id) partition p0 values less than()

The definition range of the partition must be continuous , And cannot overlap , Use values less than() To define the partition range , Define the scope from small to large .

When assigning a value to a partition field, the value range of the partition field cannot exceed values less than() Value range of .

Use values less than maxvalue To put future uncertain values into this table partition .

By time type (datetime) To partition tables, you can RANGE() Use functions in to do conversion ,

for example ：partition by range(year(create_time)),timestamp have access to unix_timestamp(‘2019-11-20 00:00:00’) conversion

create table user_range(
  id int,
  username varchar(255),
  password varchar(255),
  createDate date,
  primary key (id,createDate)
) engine=innodb
  partition by range(year(createDate))(
     partition  p2022  values  less  than(2023),
     partition  p2023  values  less  than(2024),
     partition  p2024  values  less  than(2025)  
);

Be careful ：
createDate Is a member of the Federation .** If createDate Not the primary key. ,
Just a normal field , Then the following error will be thrown during creation ：

Delete partition
alter table user_range drop partition p2022;
New zoning
alter table user_range add partition(partition p2025 values less than(2026));

 2.2 LIST Partition

grammar ： partition by list(id) partition p0 values in(1,2,3)

The partition field must be of integer type or the partition function returns an integer , The value range is through values in() To define , Out of commission maxvalue.

Suppose I have a user table , Users have gender , Now I want to store users separately according to gender ,

Men are stored in a partition , Women are stored in a partition ,SQL as follows ：

create  table user_list(
  id int,
  username varchar(255),
  password varchar(255),
  gender int,
  primary key(id, gender)
)engine=innodb
  partition by list(gender)(
     partition  man  values  in  (1),
     partition  woman  values  in  (0)
  );

 2.3 HASH Partition

Hash table partition , Determine the data contained in each partition according to the return value of a custom function

HASH The purpose of partitioning is to distribute data evenly among predefined partitions ,

Ensure that the amount of data in each partition is roughly the same . stay RANGE and LIST partition ,

You must explicitly specify in which partition a given column value or collection of column values should be stored ;

And in the HASH partition ,MySQL Do these things automatically ,

All the user has to do is specify an expression based on the column to be hashed , And the number of partitions .

Use HASH Partition to split a table , To be in CREATE TABLE Add... To the statement PARTITION BY HASH (expr),

among expr Is a field or an expression that returns an integer ; In addition, through PARTITIONS Property specifies the number of partitions ,

If not specified , The default number of partitions is 1, in addition ,HASH Partition cannot be deleted ,

So it can't be used DROP PARTITION Delete the partition .

grammar ：partition by hash(id) partitions 4

according to hash Algorithm to allocate to the partition , Set four partitions above , And according to id%4 To perform modular operations , Insert into the specified partition according to the remainder .

create table user7(id int) partition by hash(id) partitions 3;

 2.4 KEY Partition

KEY Zoning and HASH Partition similarity , however KEY Partitions support division text and BLOB Partition of all data types except ,

and HASH Partitions only support digital partitions .KEY Partitioning does not allow user-defined expressions for partitioning ,

KEY Partitions use the system provided HASH Function partition . When there is a primary key or unique index in the table ,

If you create KEY If no field is specified during partition, the system will choose the primary key column as the partition field by default ,

If there is no primary key column, the non empty unique index column is selected as the partition field .

key() Brackets can contain 0 One or more fields （ It doesn't have to be an integer type , It can be a normal field ）

create table user_key(
  id int,
  username varchar(255),
  password varchar(255),
  gender int,
  primary key(id, gender)
)engine=innodb partition by key() partitions 4;

 2.5 Multi field partitioning

 You can specify multiple fields as partition fields 
COLUMN  Zoning is  5.5  The partition function introduced , Only  RANGE COLUMN  and  LIST COLUMN  These two partitions ;
 Support shaping 、 date 、 character string ; This partitioning method is similar to  RANGE、LIST  The partition method of is very similar .
COLUMNS Vs RANGE Vs LIST  Partition ：
 The partition for date field does not need to be converted by function .
COLUMN  Partitioning supports multiple fields as partitioning keys, but does not support expressions as partitioning keys .
COLUMNS  Types of support 
 Plastic support ：tinyint、smallint、mediumint、int、bigint; I won't support it  decimal  and  float.
 Time type support ：date、datetime.
 Character types support ：char、varchar、binary、varbinary; I won't support it text、blob.

create table user1(
  id int,
  username varchar(255),
  password varchar(255),
  gender int,
  createDate date,
  primary key(id, createDate)
)engine=innodb PARTITION BY RANGE COLUMNS(createDate) (
    PARTITION p0 VALUES LESS THAN ('1990-01-01'),
    PARTITION p1 VALUES LESS THAN ('2000-01-01'),
    PARTITION p2 VALUES LESS THAN ('2010-01-01'),
    PARTITION p3 VALUES LESS THAN ('2020-01-01'),
    PARTITION p4 VALUES LESS THAN MAXVALUE
);


create table user2(
  id int,
  username varchar(255),
  password varchar(255),
  gender int,
  createDate date,
  primary key(id, createDate)
)engine=innodb PARTITION BY LIST COLUMNS(createDate) (
    PARTITION p0 VALUES IN ('1990-01-01'),
    PARTITION p1 VALUES IN ('2000-01-01'),
    PARTITION p2 VALUES IN ('2010-01-01'),
    PARTITION p3 VALUES IN ('2020-01-01')
);

 3. Common partition management commands

1. Add partition ：
alter table user add partition (partition p3 values less than (4000)); – range  Partition 
alter table user add partition (partition p3 values in (40)); – lists Partition 
2. Delete table partition （ The data will be deleted ）：
alter table user drop partition p30;
3. Delete all partitions of the table （ No loss of data ）：
alter table user_list remove partitioning;
4. Redefinition  list Partition table （ No loss of data ）：
alter table user_list partition by list(gender)(
partition man values in (1),
partition woman values in (0)
);
5. Redefinition  hash  Partition table （ No loss of data ）：
alter table user partition by hash(salary) partitions 7;
6. Merge partitions ： hold  2  Merge partitions into one , No loss of data ：
alter table user reorganize partition p1,p2 into (partition p1 values less than (30));
 Be careful ： After merging, the range is the largest 
6. Data dictionary query 
select * from information_schema.partitions
where table_schema=‘jeames’ and table_name=‘user’\G

 4. Table partition practice

 4.1 Zone Management

– Create a partition table 
create table user(id int(11) not null,name varchar(32) not null)
partition by range(id)
(
partition p0 values less than(10),
partition p1 values less than(20),
partition p2 values less than(30),
partition p3 values less than maxvalue
)

The data storage file will be split into multiple copies according to the partition

insert into user values(1,‘IT’);

insert into user values(12,‘007’);

insert into user values(22,‘jeames’);

insert into user values(50,‘TenKE’);

select * from user partition(p0);

select * from user partition(p1);

select * from user partition(p2);

select * from user partition(p3);

After adding several pieces of data, you can see that the data has been scattered in different partitions

 4.2 Conversion between ordinary table and partition table

General table to partition table statement :
ALTER TABLE students PARTITION BY KEY(sid) PARTITIONS 2;

Remove partition information
ALTER TABLE fg_pm_nbiot_cel_h_cel remove partitioning;