PG vacuum auto vacuum

Hello everyone , Today, my cousin shares with you ： PG Of  auto vacuum  Some small principle test examples .
Vacuum The role of should be well known ：

1. Delete PG MVCC A large number of dead ancestors produced under the mechanism , Provide reuse  Storage space

2. prevent transaction id wraparound overflow

3. to update visibility map , It can improve the efficiency of overlay index query

4. by Query planner  Provides the latest statistical information

There are many ways to monitor the dead ancestor , The simplest is direct query pg_stat_user_tables -> n_dead_tup

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 create table t1(id int,name varchar(200));CREATE TABLE[email protected][local:/tmp]:1992=#31111 update t1 set name ='jason' where id <= 10;UPDATE 10[email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_vacuum,last_analyze from pg_stat_user_tables where relname = 't1';relname | n_tup_ins | n_dead_tup | last_vacuum | last_analyze---------+-----------+------------+-------------+--------------t1 | 10000 | 10 | |(1 row)
  
  
  

Yes, of course extension pgstattuple Observe ： Reference resources dead_tuple_count -> dead_tuple_count

  
  
  

   
   
   [email protected][local:/tmp]:1992=#40424 create extension pgstattuple;CREATE [email protected][local:/tmp]:1992=#31111 select * from pgstattuple('t1');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+-------------- 688128 | 10000 | 609730 | 88.61 | 10 | 610 | 0.09 | 5336 | 0.78(1 row)
  
  
  

We can do it manually vacuum perhaps Trigger auto_vacuum To achieve Deletion of dead Yuanzu ：

function vacuum after , We can see n_dead_tup and dead_tuple_count Turned into 0

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 vacuum t1;[email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_vacuum,last_analyze from pg_stat_user_tables where relname = 't1';relname | n_tup_ins | n_dead_tup | last_vacuum | last_analyze---------+-----------+------------+-------------------------------+--------------t1 | 10000 | 0 | 2022-07-25 09:34:23.261923+08 |(1 row)[email protected][local:/tmp]:1992=#31111 select * from pgstattuple('t1');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+-------------- 688128 | 10000 | 609730 | 88.61 | 0 | 0 | 0 | 5976 | 0.87(1 row)
  
  
  

PG Start with version Support Auto vacuum Characteristics of ： Automatically clean up the dead Yuanzu

About auto vacuum The parameters of are as follows ：

autovacuum_naptime – For each database level Distribute worker The minimum delay time of the process The default is 1 minute
autovacuum_max_workers – vacuum Of worker Number of processes , The default is 3 individual
log_autovacuum_min_duration -- hold vacuum The operation of is greater than the specified time to write to the log , The default is 1 minute

Official vacuum threshold The formula of is ：

vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples

We are in accordance with the t1 For example ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 show autovacuum_vacuum_threshold;autovacuum_vacuum_threshold-----------------------------50(1 row)vacuum base threshold  yes  50 [email protected][local:/tmp]:1992=#31111 show autovacuum_vacuum_scale_factor;autovacuum_vacuum_scale_factor--------------------------------0.2(1 row)vacuum scale factor  yes  0.2 [email protected][local:/tmp]:1992=#31111 select reltuples from pg_class where relname = 't1';reltuples----------- 10000(1 row)t1  Tabular  vacuum base threshold  yes  10000
  
  
  

Then trigger table t1 Of auto vacuum The threshold of is ： 2050

  
  
  

   
   
   vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples = 50 + 10000*0.2 =2050 
  
  
  

Let's try to change log_autovacuum_min_duration Value , （ It is amended as follows 0 When ） Means to open all records vacuum The operation of .

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 select context from pg_settings where name ='log_autovacuum_min_duration';context---------sighup(1 row)[email protected][local:/tmp]:1992=#31111 alter system set log_autovacuum_min_duration='0';ALTER SYSTEM[email protected][local:/tmp]:1992=#31111 select pg_reload_conf();pg_reload_conf----------------t(1 row)
  
  
  

We try to update the record 2051 strip ： It has to be exceeds 2050 , To trigger .

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 update t1 set name = 'jason' where id between 5000 and 7050;UPDATE 2051
  
  
  

We observe the log ：

  
  
  

   
   
   2022-07-25 10:16:04 CST [38114]: db=,user=,app=,client= LOG: automatic vacuum of table "dbtest.public.t1": index scans: 0 pages: 0 removed, 95 remain, 95 scanned (100.00% of total) tuples: 1 removed, 10000 remain, 0 are dead but not yet removable removable cutoff: 1523544, which was 0 XIDs old when operation ended index scan not needed: 19 pages from table (20.00% of total) had 2051 dead item identifiers removed avg read rate: 0.000 MB/s, avg write rate: 36.906 MB/s buffer usage: 209 hits, 0 misses, 16 dirtied WAL usage: 56 records, 17 full page images, 145243 bytes system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
  
  
  

We can see that the log records detailed information about the table dbtest.public.t1 Conduct auto vacuum Statistical information .
We can also learn from pg_stat_user_tables Of last_autovacuum Observe the last trigger last_autovacuum Time for

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't1';relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ---------+-----------+------------+-------------------------------+-------------------------------t1 | 10000 | 0 | 2022-07-25 10:16:04.974528+08 | 2022-07-25 10:10:04.931326+08(1 row)
  
  
  

The official also gave auto analyze Formula ：

analyze threshold = analyze base threshold + analyze scale factor * number of tuples

Again, we follow t1 Table examples ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 show autovacuum_analyze_threshold;autovacuum_analyze_threshold------------------------------50(1 row)analyze base threshold  yes  50[email protected][local:/tmp]:1992=#31111 show autovacuum_analyze_scale_factor;autovacuum_analyze_scale_factor---------------------------------0.1(1 row)analyze scale factor  yes  0.1[email protected][local:/tmp]:1992=#31111 select reltuples from pg_class where relname = 't1';reltuples----------- 10000(1 row)t1  Tabular  vacuum base threshold  yes  10000
  
  
  

Then trigger table t1 Of auto analyze The threshold of is ： 1050

  
  
  

   
   
   vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples = 50 + 10000*0.1 =1050 
  
  
  

We try to update the record 1051 strip ： It has to be exceeds 1050 , To trigger

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 update t1 set name ='jason2022' where id between 1000 and 2050;UPDATE 1051
  
  
  

We see about the table t1 Of last_autoanalyze Has been updated ：2022-07-25 10:44:05.285096+08

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't1';relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ---------+-----------+------------+-------------------------------+-------------------------------t1 | 10000 | 1051 | 2022-07-25 10:16:04.974528+08 | 2022-07-25 10:44:05.285096+08(1 row)
  
  
  

We observe the log of the system ：

  
  
  

   
   
   2022-07-25 10:44:05 CST [42207]: db=,user=,app=,client= LOG: automatic analyze of table "dbtest.public.t1" avg read rate: 0.000 MB/s, avg write rate: 6.875 MB/s buffer usage: 332 hits, 0 misses, 22 dirtied system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
  
  
  

Next, we are thinking about a problem ： insert Whether the statement of will also Trigger auto vacuum ?

The answer given in the official documents is yes ： The table is also vacuumed if the number of tuples inserted since the last vacuum has exceeded the defined insert threshold.

  
  
  

   
   
   vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
  
  
  

According to the official calculation formula , Let's follow the table t1 To calculate ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 show autovacuum_vacuum_insert_threshold;autovacuum_vacuum_insert_threshold------------------------------------1000(1 row)[email protected][local:/tmp]:1992=#31111 show autovacuum_vacuum_insert_scale_factor;autovacuum_vacuum_insert_scale_factor---------------------------------------0.2(1 row)[email protected][local:/tmp]:1992=#31111 select reltuples from pg_class where relname = 't1';reltuples----------- 10000(1 row)vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples = 1000 + (0.2 * 10000) = 3000 
  
  
  

According to the official Formula , We can see that surface t1 Of insert The threshold of is 3000, We try to insert 3001 Bar record ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 insert into t1 select id, md5(id::varchar) from generate_series(1,3001) as id;INSERT 0 3001
  
  
  

We can see from the log that , At the same time, it triggers auto vacuum and auto analyze The operation of

tuples: 1051 removed, Clean it up 1051 The number of dead Yuanzu Our last case was used to update manually Automatic triggering analyze The number of Yuanzu

  
  
  

   
   
   2022-07-25 11:00:05 CST [44722]: db=,user=,app=,client= LOG: automatic vacuum of table "dbtest.public.t1": index scans: 0 pages: 0 removed, 108 remain, 73 scanned (67.59% of total) tuples: 1051 removed, 11792 remain, 0 are dead but not yet removable removable cutoff: 1523547, which was 0 XIDs old when operation ended index scan not needed: 10 pages from table (9.26% of total) had 1050 dead item identifiers removed avg read rate: 0.000 MB/s, avg write rate: 43.149 MB/s buffer usage: 165 hits, 0 misses, 17 dirtied WAL usage: 64 records, 18 full page images, 84882 bytes system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s2022-07-25 11:00:05 CST [44722]: db=,user=,app=,client= LOG: automatic analyze of table "dbtest.public.t1" avg read rate: 0.279 MB/s, avg write rate: 1.395 MB/s buffer usage: 340 hits, 1 misses, 5 dirtied system usage: CPU: user: 0.02 s, system: 0.00 s, elapsed: 0.02 s
  
  
  

Why? insert The operation will also trigger vacuum The operation of ？

The answer given in the official document is

Such vacuums may allow portions of the table to be marked as all visible and also allow tuples to be frozen, which can reduce the work required in subsequent vacuums.

Translated into vernacular is , For the next time vacuum, Reduce the burden .

Next , Let's see whether the temporary table and partitioned table will trigger auto vacuum ?

Test temporary table ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 create temporary table t_tmp (id int, name varchar(200));CREATE TABLE[email protected][local:/tmp]:1992=#31111 insert into t_tmp select id, md5(id::varchar) from generate_series(1,10000) as id;INSERT 0 10000
  
  
  

Calculated according to the previous formula Trigger auto vacuum The threshold of is 2050, Let's try to update ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 update t_tmp set name ='jason2022' where id between 1 and 2051;UPDATE 2051
  
  
  

We found through observation that , Temporary tables do not trigger auto vacuum The operation of

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_tmp';relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze---------+-----------+------------+-----------------+------------------t_tmp | 10000 | 2051 | |(1 row)
  
  
  

Let's trigger it manually vacuum Your order is ok

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 select relname,n_tup_ins,n_dead_tup,last_vacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_tmp';relname | n_tup_ins | n_dead_tup | last_vacuum | last_autoanalyze---------+-----------+------------+-------------------------------+------------------t_tmp | 10000 | 0 | 2022-07-25 11:36:17.662935+08 |(1 row)
  
  
  

Let's test the partition table again ： We create a hash The partition table ： 4 individual hash The partition

  
  
  

   
   
   [email protected][local:/tmp]:1992=#31111 create table t_person(id int, name varchar(200)) partition by hash(id);CREATE TABLE[email protected][local:/tmp]:1992=#31111 create table t_person_0 partition of t_person for values with (MODULUS 4,REMAINDER 0);CREATE TABLE[email protected][local:/tmp]:1992=#31111 create table t_person_1 partition of t_person for values with (MODULUS 4,REMAINDER 1);CREATE TABLE[email protected][local:/tmp]:1992=#31111 create table t_person_2 partition of t_person for values with (MODULUS 4,REMAINDER 2);CREATE TABLE[email protected][local:/tmp]:1992=#31111 create table t_person_3 partition of t_person for values with (MODULUS 4,REMAINDER 3);CREATE TABLE
  
  
  

We try to insert ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#65714 insert into t_person select id, md5(id::varchar) from generate_series(1,40000) as id;INSERT 0 40000
  
  
  

Let's try , to update 25% The data of ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#65714 update t_person set name ='jason2022' where id <= 10000;UPDATE 10000
  
  
  

Let's check the partition table , It is found that all records are empty

  
  
  

   
   
   [email protected][local:/tmp]:1992=#65714 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_person';relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze----------+-----------+------------+-----------------+------------------t_person | 0 | 0 | |(1 row)
  
  
  

We are checking 4 Sub tables of partitions ： It also triggered auto vacuum

  
  
  

   
   
   [email protected][local:/tmp]:1992=#65714 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_person_0'; relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ------------+-----------+------------+-------------------------------+-------------------------------t_person_0 | 10034 | 0 | 2022-07-25 13:39:07.422698+08 | 2022-07-25 13:39:07.448201+08(1 row)[email protected][local:/tmp]:1992=#65714 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_person_1'; relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ------------+-----------+------------+------------------------------+-------------------------------t_person_1 | 10067 | 0 | 2022-07-25 13:39:07.45636+08 | 2022-07-25 13:39:07.475879+08(1 row)[email protected][local:/tmp]:1992=#65714 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_person_2'; relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ------------+-----------+------------+-------------------------------+-------------------------------t_person_2 | 10031 | 0 | 2022-07-25 13:39:07.482265+08 | 2022-07-25 13:39:07.502235+08(1 row)[email protected][local:/tmp]:1992=#65714 select relname,n_tup_ins,n_dead_tup,last_autovacuum,last_autoanalyze from pg_stat_user_tables where relname = 't_person_3'; relname | n_tup_ins | n_dead_tup | last_autovacuum | last_autoanalyze ------------+-----------+------------+-------------------------------+-------------------------------t_person_3 | 9868 | 0 | 2022-07-25 13:39:07.504214+08 | 2022-07-25 13:39:07.521341+08(1 row)
  
  
  

At present, our conclusion through experiments is ： The partition parent table will not have auto vacuum The record of , The sub table of the partition triggers auto vacuum The process of

About vacuum Of IO Adjust the ：

because vacuum When the process starts , Yes IO The impact of consumption is very large , postgres Related parameters of autovacuum_vacuum_cost_delay and autovacuum_vacuum_cost_limit To balance the system IO Parameters consumed .
autovacuum_vacuum_cost_limit ： meanwhile running autovacuum workers The upper limit of , Beyond this value Then enter the process sleep state The default is 200

autovacuum_vacuum_cost_delay： When the process reaches the upper limit （autovacuum_vacuum_cost_limit）, process Will be carried out in sleep Time for The default is 2ms

In theory, , autovacuum_vacuum_cost_limit If the setting is too large , For busy systems , There will be IO Bottleneck .
autovacuum_vacuum_cost_delay If the setting is too large , Will lead to vacuum A large number of threads are waiting , So it reduces vacuum The efficiency of .

According to the characteristics of your business system and database server CPU,IO Specific performance , To decide whether to proceed vacuum Parameter adjustment of .

Finally, let's summarize ：

1) monitor dead tuple , We can use pg_stat_user_tables and extension pgstattuple
2) Automatic triggering auto vacuum Calculation formula ： insert Operation will also trigger auto vacuum

  
  
  

   
   
   vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuplesvacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuplesanalyze threshold = analyze base threshold + analyze scale factor * number of tuples 
  
  
  

3) The temporary table will not trigger auto vacuum, The sub partition table of the partition table will be triggered separately auto vacuum
4)Vacuum Of IO Adjust the parameters ：autovacuum_vacuum_cost_limit and autovacuum_vacuum_cost_delay

Have a fun !