Introduction to automatic partition management of gaussdb (DWS)

Abstract

​ For partitioned tables whose partitions are listed as time , The automatic partition management function can automatically create new partitions and delete expired partitions , Reduce the maintenance cost of partition tables , Improve query performance . The following will discuss the role of partition automatic management 、 usage 、 Principle and other aspects .

1. Partition automatic management function

​ To facilitate the query and maintenance of data , Customers usually use partitioned tables that are partitioned as time to store time-related data , For example, e-commerce order information 、 Real time data collected by the Internet of things . When these time-dependent data are imported into the partitioned table , It is necessary to ensure that the partition table has partitions corresponding to the time , Because ordinary partition tables do not automatically create new partitions or delete expired partitions , Therefore, maintenance personnel need to regularly create new partitions and delete expired partitions , Increased operation and maintenance costs .

​ To solve the above problems ,GaussDB(DWS) The automatic partition management feature is introduced . Users can set table level parameters period、ttl Enable automatic partition management , The partition table can automatically create new partitions and delete expired partitions , Reduce the maintenance cost of partition tables , Improve query performance . among ,period Indicates the time range of the new partition , It also indicates the cycle of automatically creating new partitions and automatically deleting expired partitions ;ttl Indicates the partition expiration time .period and ttl Values are Interval type , for example ’1 hour’、‘1 day’、‘1 week’、‘1 month’、‘1 year’、 '1 month 2 days 3 hours’ etc. .

1.1 Automatically create new partitions

​ Partition automatic management every period The partition will be created automatically after the time , One or more time ranges are created at a time period New partition for , To advance the maximum partition boundary time , Ensure that it is greater than nowTime+30*period. Because every time you create a partition , Both dynamically create reserved partitions for future time , So once a new partition is created automatically , Can guarantee in the future 30 individual period Within a period of time , The import of real-time data fails because there is no corresponding partition .

1.2 Automatically delete expired partitions

​ The boundary time is earlier than nowTime-ttl The partition of is considered to be out of date . Partition automatic management every period Time will be traversed to detect all partitions , And delete the expired partition , If all partitions are expired , Keep a partition , and truncate This table .

2. Partition auto management usage

2.1 Partition management function constraints

​ When using the partition management function , The following constraints need to be met ：

1） Not supported on minicomputers 、 Speed up the cluster 、 On a stand-alone cluster .

2） Does not support the 8.1.3 Used in versions below .

3） Only row storage range partition tables are supported 、 List and save the range partition table 、 Time schedule and hot and cold meters .

4） The partition key is unique and the type only supports timestamp、timestamptz、date type .

5） Does not support the presence of maxvalue Partition .

6）(nowTime - boundaryTime) / period It needs to be less than the maximum number of partitions , among nowTime For the current time ,boundaryTime Is the earliest partition boundary time in the existing partition .

7）period、ttl The value range is 1 hour ~ 100 years. in addition , Compatible with Teradata or MySQL In the database of , The partition key type is date when ,period Not less than 1 day.

8） Table level parameters ttl It is not supported to exist alone , Must be set in advance or at the same time period, And must be greater than or equal to period.

9. During the online capacity expansion of the cluster , Automatically adding partitions will fail , But because every time the partition is added , Enough partitions are reserved , So it doesn't affect the use of .

2.2 Enable the partition management function

​ Partition management functions are related to table level parameters period、ttl The binding of , As long as the table level parameters are set successfully period, That is, the function of automatically creating new partitions is enabled ; Successfully set table level parameters ttl, That is, the function of automatically deleting expired partitions is enabled . The first time a partition is automatically created or deleted is set to period or ttl after 30 second .

​ There are two ways to enable partition management , As follows ：

1） When creating a table, specify period、ttl

​ This method is applicable to creating partition management tables . There are two syntax for creating a partition management table , One is to specify partitions when creating tables , The other is to create tables without specifying partitions .

​ When creating a partition management table, if you specify a partition , The syntax rules are the same as those for creating a common partition table , The only difference is that table level parameters are specified period、ttl, Specific examples are as follows . In this example , The partition expires on 7 days, The time range of the new partition and the task cycle of the automatic partition are 1 day.

CREATE TABLE CPU1(	    id integer,		    IP text,	    time timestamp) with (TTL='7 days',PERIOD='1 day')partition by range(time)(	    PARTITION P1 VALUES LESS THAN('2022-01-05 16:32:45'),    PARTITION P2 VALUES LESS THAN('2022-01-06 16:56:12'));

​ When creating a partition management table, you can specify only the partition key and not the partition , Two default partitions are created , The partition time range for both default partitions is period. among , The boundary time of the first default partition is the first integer hour greater than the current time / All day / All week / whole month / The whole year , The exact time you choose depends on period The largest unit of ; The boundary time of the second default partition is the boundary time of the first partition plus period. Suppose the current time is 2022-02-17 16:32:45, The partition boundary of the first default partition in each case is selected in the following table ：

period	period The unit is the largest	The partition boundary of the first default partition
1 hour	Hour	2022-02-17 17:00:00
1 day	Day	2022-02-18 00:00:00
1 month	Month	2022-03-01 00:00:00
13 months	Year	2023-01-01 00:00:00

​ A specific example of creating a table without specifying a partition is as follows ：

CREATE TABLE CPU2(    id integer,	    IP text,	    time timestamp) with (TTL='7 days',PERIOD='1 day')partition by range(time);

2） Use alter table set How to set period、ttl

​ This method is applicable to adding partition management functions to a common partition table that meets partition management constraints .

​ hypothesis cpu3 A table is a common partition table that satisfies the partition management constraints , An example of adding partition management functions to this table is as follows ：

--  At the same time, enable the automatic creation and deletion of partitions ALTER TABLE cpu3 SET (PERIOD='1 day',TTL='7 days');--  Only enable automatic partition creation ALTER TABLE cpu3 SET (PERIOD='1 day');--  Only enable the auto delete partition function , If the automatic partition creation function is not enabled in advance , Failed to open ALTER TABLE cpu3 SET (TTL='7 days');

2.3 Modify the partition management function

​ The main function of modifying partition management is to modify period and ttl, Can pass alter table set The way to modify .

​ hypothesis cpu4 Table is a partition management table , Modify it period and ttl Examples are as follows ：

--  At the same time to modify periodALTER TABLE cpu4 SET (TTL='10 days',PERIOD='2 days');

2.4 Turn off the partition management function

​ Use alter table reset Can delete table level parameters period、ttl, Then turn off the corresponding partition management function . It should be noted that , Cannot exist ttl Under the circumstances , Delete separately period. in addition , Timeline does not support alter table reset.
​ hypothesis cpu5 A table is a sheet with period and ttl Partition management table , An example of turning off the partition management function is as follows ：

--  Turn off the automatic creation and deletion of partitions ALTER TABLE cpu5 RESET (PERIOD,TTL);--  Only turn off the auto delete partition function ALTER TABLE cpu5 RESET (TTL);--  Only turn off automatic partition creation , If the table has ttl Parameters , Then the shutdown fails ALTER TABLE cpu5 RESET (PERIOD);

3. Principle of automatic partition management

3.1 The basic principle

​ The implementation of partition management relies on pg_task Automatically schedule tasks , Setting the period/ttl when , towards scheduler.pg_task Self incrementing is inserted in the table / Self reducing partition management tasks , The task content of the self adding partition task is proc_add_partition(relname, period) function , The self decreasing partition task is proc_drop_partition(relname, ttl) function , The call cycle of both tasks is period, The first execution time is after the task insertion time 30 second .

​ Use administrator privileges , You can see scheduler.pg_task Task specific information in the table , For example, task content (what Field )、 Task execution cycle (interval Field )、 The last time the task was successfully executed (actual_end_time Field )、 Task status (task_status Field ). When the automatic scheduling task fails to execute , Users can copy scheduler.pg_task Partition management tasks in the table what Field , Then perform the task manually . see scheduler.pg_task The table and the example of manually executing the auto increment partition task are as follows ：

my_database=# SELECT what,interval FROM scheduler.pg_task;                             what                             | interval--------------------------------------------------------------+---------- call proc_add_partition('public.cpu1', interval '1 day');    | 1 day call proc_drop_partition('public.cpu1', interval '7 days');  | 1 day(2 rows)my_database=# call proc_add_partition('public.cpu1', interval '1 day'); proc_add_partition--------------------(1 row)

3.2 proc_add_partition(relname regclass, boundaries_interval interval)

​ This function is used to automatically add partitions , The specific implementation process is shown in the following figure . Function runtime , On the basis of the partition boundary , Create multiple time ranges of boundaries_interval New partition for , Until the new boundary The distance from the current time is greater than 29 individual boundaries_interval, Then create an additional partition , Ensure that the function runs , A new partition will be created .

3.3 proc_drop_partition (relname regclass, older_than interval)

​ This function is used to automatically delete expired partitions . Function runtime , Will traverse all partitions of the partition table , And delete boundary Before (now_time - older_than) The partition ; If all partitions meet the deletion criteria , Keep a partition , and truncate This table .