Automatically create partition tables by time and month in PostgreSQL

Preface

In my work, I encountered ten million level single table data query optimization , Bloggers think of table partition for the first time , The process of solving the problem is hereby recorded , Avoid digging holes .
Database table partition divides a large physical table into several small physical tables , And these small physical tables can be logically used as a table .
PostgreSQL Partition the table in the database , And 11g Previous Oracle similar , You need to create partition tables manually 、 Index, etc. . By inheritance , For example, by time , Create a table partition every month , Data is recorded in the corresponding partition .

1. Create the main table

Create main table structure , The name of the table three_height_data_value, The name of the time field : data_time

CREATE TABLE "public"."three_height_data_value" (
  "data_id" varchar(64) COLLATE "pg_catalog"."default" NOT NULL,
  "data_time" timestamp(6) NOT NULL,
  "data_value" float8 NOT NULL
);

2. Create stored procedure

Create stored procedure , Usage dynamics SQL, Automate the operation and maintenance of large table partitions .

CREATE OR REPLACE FUNCTION auto_insert_into_three_height_data_value()
  RETURNS trigger AS
$BODY$
DECLARE
    time_column_name 	text ;			--  The name of the time field used for partitioning in the parent table [ Must first initialize !!]
    curMM 		varchar(6);		-- 'YYYYMM' String , Used as the suffix of partition sub table 
    isExist 		boolean;		--  Partition sub table , Does it already exist 
    startTime 		text;
    endTime		text;
    strSQL  		text;
    
BEGIN
    --  Before calling , Must first initialize ( Time field name ):time_column_name [ Get... Directly from the call parameters !!]
    time_column_name := TG_ARGV[0];
   
    --  Determine the corresponding partition table   Does it already exist ?
    EXECUTE 'SELECT $1.'||time_column_name INTO strSQL USING NEW;
    curMM := to_char( strSQL::timestamp , 'YYYYMM' );
    select count(*) INTO isExist from pg_class where relname = (TG_RELNAME||'_'||curMM);
 
    --  If it does not exist ,  Before insertion   Create a sub partition first 
    IF ( isExist = false ) THEN  
        --  Create a sub partition table 
        startTime := curMM||'01 00:00:00';
        endTime := to_char( startTime::timestamp + interval '1 month', 'YYYY-MM-DD HH24:MI:SS');
        strSQL := 'CREATE TABLE IF NOT EXISTS '||TG_RELNAME||'_'||curMM||
                  ' ( CHECK('||time_column_name||'>='''|| startTime ||''' AND '
                             ||time_column_name||'< '''|| endTime ||''' )
                          ) INHERITS ('||TG_RELNAME||') ;'  ;  
        EXECUTE strSQL;
 
        --  Create index 
        strSQL := 'CREATE INDEX '||TG_RELNAME||'_'||curMM||'_INDEX_'||time_column_name||' ON '
                  ||TG_RELNAME||'_'||curMM||' ('||time_column_name||');' ;
        EXECUTE strSQL;
       
    END IF;
 
    --  Insert data into sub partition !
    strSQL := 'INSERT INTO '||TG_RELNAME||'_'||curMM||' SELECT $1.*' ;
    EXECUTE strSQL USING NEW;
    RETURN NULL; 
END
$BODY$
  LANGUAGE plpgsql;

explain :
(1) The code uses TG_ARGV[0] To get the parameters passed in during the call : The name of the time field used for the partition .

(2) In the code , Through built-in parameters TG_RELNAME Get the table name of the parent table .

(3) First, according to the insertion time , Judge whether the corresponding partition table exists ? If exist , Directly insert the corresponding partition sub table

(4) If the partition table does not exist , First create partitioned sub tables and indexes , Then insert the data into the created sub table .

Above code , stay PostgreSQL v9.4 Debugging through . Theoretically , v8.4 All of the above are supported .

3. Create trigger

CREATE TRIGGER insert_three_height_data_value_trigger
  BEFORE INSERT
  ON three_height_data_value
  FOR EACH ROW
  EXECUTE PROCEDURE auto_insert_into_three_height_data_value('data_time');

4. The pit of tread

（1）constraint_exclusion Property settings

Make sure postgresql.conf Medium constraint_exclusion Configuration item is not disable . This is very important , If this parameter item is disable, Query performance based on partitioned tables cannot be optimized , Even lower performance than using indexes directly without using partitioned tables .

（2）linux How to execute postgresql Of sql Script

Method 1 ： First, through psql Connect to the corresponding db：

psql -d db1 -U userA

Then enter the password , After entering the database , Input ：

\i /pathA/xxx.sql

Method 2 ： Directly through psql Command execution SQL file

In this way, there is no need to log in to the database first , Just use one command

psql -d db1 -U userA -f /pathA/xxx.sql

（3） How to query table partitions

– Query the specified partition table

SELECT nmsp_parent.nspname AS parent_schema , parent.relname AS parent , nmsp_child.nspname AS child , child.relname AS child_schema FROM pg_inherits JOIN pg_class parent ON pg_inherits.inhparent = parent.oid JOIN pg_class child ON pg_inherits.inhrelid = child.oid JOIN pg_namespace nmsp_parent ON nmsp_parent.oid = parent.relnamespace JOIN pg_namespace nmsp_child ON nmsp_child.oid = child.relnamespace WHERE parent.relname = 'plc_import_data_value';

Query all zone tables

SELECT nspname , relname ,COUNT(*) AS partition_num FROM pg_class c , pg_namespace n , pg_inherits i WHERE c.oid = i.inhparent AND c.relnamespace = n.oid AND c.relhassubclass AND c.relkind = 'r' GROUP BY 1,2 ORDER BY partition_num DESC;

If it helps , please “ give the thumbs-up ”、“ Focus on ”、“ Collection ” Ha ！