PG free space map & visibility map

Hello everyone , Today, my cousin shares with you PG Of Free Space Map and Visibility Map.

You can see this from the name 2 The meaning of concepts ：

FSM: Every heap Tables and indexes There will be a corresponding FSM File to track the available space of the object .
VM: Every heap The object of the table There will be a corresponding VM Document tracking record ：

VISIBILITYMAP_ALL_VISIBLE: Mark what page All Yuanzu pairs contained in it active transaction Visible .
VISIBILITYMAP_ALL_FROZEN: Mark what page All the ancestors in it are frozen The state of

Let's simply create a table ： We can see FSM and VM 2 Files will be automatically generated ,

The naming rules are as follows ： oid_fsm, oid_vm

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 create table t_vm_fsm (id int, name varchar(200));CREATE TABLE[email protected][local:/tmp]:1992=#32679 select oid,relname from pg_class where relname = 't_vm_fsm'; oid | relname -------+----------33240 | t_vm_fsm(1 row)[email protected][local:/tmp]:1992=#32679 insert into t_vm_fsm values (generate_series(1,10000),'hello world!');INSERT 0 10000INFRA [postgres[email protected] 32963]# ls -lhtr | grep 33240-rw------- 1 postgres postgres 24K Jul 26 14:54 33240_fsm-rw------- 1 postgres postgres 512K Jul 26 14:54 33240-rw------- 1 postgres postgres 8.0K Jul 26 14:54 33240_vm
  
  
  

oid_fsm and oid_vm yes 2 individual Binary files , You can't directly view the contents . But the official offer 2 individual extension Check it out. Contents of binary files

pg_freespacemap It's for viewing FSM Recorded in the document blkno Block number and avi free space The size of the relationship .

Let's install it pg_freespacemap This extension:

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 create extension pg_freespacemap;CREATE EXTENSION
  
  
  

It contains a function Of 2 individual overload Methods

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 \df pg_freespace List of functionsSchema | Name | Result data type | Argument data types | Type--------+--------------+------------------+----------------------------------------------------+------public | pg_freespace | smallint | regclass, bigint | funcpublic | pg_freespace | SETOF record | rel regclass, OUT blkno bigint, OUT avail smallint | func(2 rows)
  
  
  

Let's check the table t_vm_fsm Size ： 512KB, default page Of Size is 8K, therefore There will be 512/8 = 64 individual page ( Or is it block)

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 select pg_size_pretty(pg_relation_size('t_vm_fsm'));pg_size_pretty----------------512 kB(1 row)
  
  
  

We can use functions pg_freespace Check it out. 64 individual block Usage situation , We can see 0-62 Number block in （ Note that the subscript is from 0 At the beginning ） avail space yes 0 , That is to say, there is no available space . 63 Number block Yes, there is 2496 byte Space. .

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 SELECT * FROM pg_freespace('t_vm_fsm');blkno | avail-------+------- 0 | 0 1 | 0 2 | 0 ... ... ... 62 | 0 63 | 2496(64 rows)
  
  
  

If you want to check alone 63 Number block Of AVI Space size of , Sure Pass in 2 Parameters

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 SELECT * FROM pg_freespace('t_vm_fsm',63);pg_freespace-------------- 2496(1 row)
  
  
  

If we want to set the watch now , Do some delete operation , Check the space size again ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 delete from t_vm_fsm where id < 1000;DELETE 999
  
  
  

Because the cardinality of our table is 10000 , autovacuum_vacuum_threshold yes 50 , autovacuum_vacuum_scale_factor yes 0.2

So trigger auto_vacuumn The threshold of is 50 + 10000 * 0.2 = 2050,

999 Deletion of data , The threshold is not reached , So it doesn't trigger auto vacuum

Let's look at... Again pg_freespace ： Nothing has changed

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 SELECT * FROM pg_freespace('t_vm_fsm');blkno | avail-------+------- 0 | 0 1 | 0 2 | 0 ... ... ... 62 | 0 63 | 2496(64 rows)
  
  
  

Let's do it manually vacuum table The operation of ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 vacuum t_vm_fsm;VACUUM
  
  
  

Look again pg_freespace ： We can find out block 0-6 Number Reusable space has been released

  
  
  

   
   
   [email protected][local:/tmp]:1992=#32679 select * from (SELECT * FROM pg_freespace('t_vm_fsm')) fsm where fsm.avail > 0;blkno | avail-------+------- 0 | 8160 1 | 8160 2 | 8160 3 | 8160 4 | 8160 5 | 8160 6 | 2720 63 | 2496(8 rows)
  
  
  

Let's run it again vacuum full , Reclaim physical space , Note that this is for locking the watch , Production needs to be disabled ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#48460 vacuum full t_vm_fsm;VACUUM
  
  
  

Observe again ： page Has been physically restructured , Previous 64 individual block Turned into 58 individual block ,

Physical storage from 512 kB Down to 464kB

  
  
  

   
   
   [email protected][local:/tmp]:1992=#48460 select * from (SELECT * FROM pg_freespace('t_vm_fsm')) fsm where fsm.avail > 0;blkno | avail-------+-------(0 rows)[email protected][local:/tmp]:1992=#48460 SELECT count(1) FROM pg_freespace('t_vm_fsm');count------- 58(1 row)[email protected][local:/tmp]:1992=#48460 select pg_size_pretty(pg_relation_size('t_vm_fsm'));pg_size_pretty----------------464 kB(1 row)
  
  
  

From a code perspective FSM Use data structure Binary tree to maintain , Students interested in algorithms can have a look :
src/backend/storage/freespace/README

  
  
  

   
   
   FSM page structure------------------Within each FSM page, we use a binary tree structure where leaf nodes storethe amount of free space on heap pages (or lower level FSM pages, see"Higher-level structure" below), with one leaf node per heap page. A non-leafnode stores the max amount of free space on any of its children.For example: 44 23 4 0 2 <- This level represents heap pages
  
  
  

Now let's move on to Visibility Map ：

The official document defines The visibility map stores two bits per heap page. For every one page perhaps block Store a 2 bit The tag

a) first bit It's the mark VISIBILITYMAP_ALL_VISIBLE, If one page All of the tuple All are all visible Of , Then mark
b) the second bit It's the mark VISIBILITYMAP_ALL_FROZEN, If one page All of the tuple All are forzen Of , Then mark

Visibility Map Of 2 The main use is ：

1. Trigger Index only scan The interview of , To speed up the SQL Execution efficiency
2. Speed up vacuum Efficiency of execution

We can go through extension : pg_visibility Let's observe ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 create extension pg_visibility;CREATE [email protected][local:/tmp]:1992=#80334 create table t_vm (id int , name varchar(200));CREATE TABLE[email protected][local:/tmp]:1992=#80334 insert into t_vm values(generate_series(1,10000),'hello VM !');INSERT 0 10000[email protected][local:/tmp]:1992=#80334 create index concurrently idx_name_vm on t_vm(name);CREATE INDEX
  
  
  

We can see , The current table has not been modified , Delete and other operations , So in VM The ancestors of the Yuan Dynasty are visible .

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 select count(0) from pg_visibility('t_vm') ;count------- 55(1 row)[email protected][local:/tmp]:1992=#80334 select count(0) from pg_visibility('t_vm') where all_visible = 'f' ;count------- 0(1 row)[email protected][local:/tmp]:1992=#80334 select all_visible, count(1) from pg_visibility('t_vm') group by all_visible ;all_visible | count-------------+-------t | 55(1 row)
  
  
  

At this time, we run a SQL You can go index only scan Implementation plan of ：

Several important information indicators ：
Index Only Scan using idx_name_vm on t_vm
Heap Fetches: 0

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 explain(analyze) select name from t_vm where name = 'test'; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------Index Only Scan using idx_name_vm on t_vm (cost=0.29..4.30 rows=1 width=11) (actual time=0.016..0.017 rows=0 loops=1) Index Cond: (name = 'test'::text) Heap Fetches: 0Planning Time: 0.470 msExecution Time: 0.045 ms(5 rows)
  
  
  

Now we want to update some data , Be careful not to trigger the updated data auto vacuum The process of , because vacuum At the end of the process , Will update VM

Because the cardinality of our table is 10000 , autovacuum_vacuum_threshold yes 50 , autovacuum_vacuum_scale_factor yes 0.2

So trigger auto_vacuumn The threshold of is 50 + 10000 * 0.2 = 2050,

1000 Update of data , Not trigger auto vacuum

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 update t_vm set name = 'test' where id <= 1000;UPDATE 1000
  
  
  

We can see Yes 12 individual page (block ) After performing the update operation , The tag has been updated to f

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 select all_visible, count(1) from pg_visibility('t_vm') group by all_visible ;all_visible | count-------------+-------f | 12t | 48(2 rows)
  
  
  

Let's do it again SQL： You can see Heap Fetches: 1000 Turned into 1000

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 explain(analyze) select name from t_vm where name = 'test'; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------Index Only Scan using idx_name_vm on t_vm (cost=0.29..4.30 rows=1 width=11) (actual time=0.030..0.179 rows=1000 loops=1) Index Cond: (name = 'test'::text) Heap Fetches: 1000Planning Time: 0.065 msExecution Time: 0.225 ms(5 rows)
  
  
  

We manually vacuum Look at the watch t_vm :

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 vacuum t_vm;VACUUM
  
  
  

Observe again VM The mark of a ： All marked as visibility

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 select all_visible, count(1) from pg_visibility('t_vm') group by all_visible ;all_visible | count-------------+-------t | 60(1 row)
  
  
  

Look again SQL Implementation plan of ： Heap Fetches: 0 Turned into 0

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 explain(analyze) select name from t_vm where name = 'test'; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------Index Only Scan using idx_name_vm on t_vm (cost=0.29..4.30 rows=1 width=11) (actual time=0.020..0.093 rows=1000 loops=1) Index Cond: (name = 'test'::text) Heap Fetches: 0Planning Time: 0.120 msExecution Time: 0.147 ms(5 rows)
  
  
  

Let's keep going VISIBILITYMAP_ALL_FROZEN This marker bit ：

Let's delete 500 Bar record ：

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 delete from t_vm where id <= 500;DELETE 500
  
  
  

We can observe frozen block Information about ： blkno [0-4] altogether 5 Block Marked as all_frozen = t

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 select blkno, all_frozen from pg_visibility('t_vm') where all_frozen = 't';blkno | all_frozen-------+------------ 0 | t 1 | t 2 | t 3 | t 4 | t(5 rows)[email protected][local:/tmp]:1992=#80334 select * from pg_visibility_map_summary('t_vm');all_visible | all_frozen-------------+------------ 57 | 5(1 row)
  
  
  

We can go through extension pageinspect Check it out. Marked as forzen Of blkno [0-4] The contents are empty

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 create extension pageinspect;CREATE [email protected][local:/tmp]:1992=#80334 select * from heap_page_items(get_raw_page('t_vm',0));lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_data----+--------+----------+--------+--------+--------+----------+--------+-------------+------------+--------+--------+-------+-------- 1 | 0 | 0 | 0 | | | | | | | | | |(1 row)[email protected][local:/tmp]:1992=#80334 select * from heap_page_items(get_raw_page('t_vm',4));lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_data----+--------+----------+--------+--------+--------+----------+--------+-------------+------------+--------+--------+-------+-------- 1 | 0 | 0 | 0 | | | | | | | | | |(1 row)
  
  
  

natural block The information of the page is as follows blkno by 5 Number Page information

  
  
  

   
   
   [email protected][local:/tmp]:1992=#80334 select * from heap_page_items(get_raw_page('t_vm',5)) where t_data is not null limit 10;lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_data ----+--------+----------+--------+---------+--------+----------+--------+-------------+------------+--------+--------+-------+----------------------------------76 | 8152 | 1 | 39 | 1523617 | 0 | 0 | (5,76) | 2 | 2306 | 24 | | | \xe90300001768656c6c6f20564d202177 | 8112 | 1 | 39 | 1523617 | 0 | 0 | (5,77) | 2 | 2306 | 24 | | | \xea0300001768656c6c6f20564d202178 | 8072 | 1 | 39 | 1523617 | 0 | 0 | (5,78) | 2 | 2306 | 24 | | | \xeb0300001768656c6c6f20564d202179 | 8032 | 1 | 39 | 1523617 | 0 | 0 | (5,79) | 2 | 2306 | 24 | | | \xec0300001768656c6c6f20564d202180 | 7992 | 1 | 39 | 1523617 | 0 | 0 | (5,80) | 2 | 2306 | 24 | | | \xed0300001768656c6c6f20564d202181 | 7952 | 1 | 39 | 1523617 | 0 | 0 | (5,81) | 2 | 2306 | 24 | | | \xee0300001768656c6c6f20564d202182 | 7912 | 1 | 39 | 1523617 | 0 | 0 | (5,82) | 2 | 2306 | 24 | | | \xef0300001768656c6c6f20564d202183 | 7872 | 1 | 39 | 1523617 | 0 | 0 | (5,83) | 2 | 2306 | 24 | | | \xf00300001768656c6c6f20564d202184 | 7832 | 1 | 39 | 1523617 | 0 | 0 | (5,84) | 2 | 2306 | 24 | | | \xf10300001768656c6c6f20564d202185 | 7792 | 1 | 39 | 1523617 | 0 | 0 | (5,85) | 2 | 2306 | 24 | | | \xf20300001768656c6c6f20564d2021(10 rows)
  
  
  

Finally, let's summarize ：

Free Space Map： Record and track each （blk or page） Free space information , The naming form of the file xxx_fsm, Can pass extension pg_freespacemap See specific details

Visibility Map： Record and track each （blk or page） Of VISIBILITYMAP_ALL_VISIBLE and VISIBILITYMAP_ALL_FROZEN Information . The naming form of the file xxx_vm, Can pass extension pg_visibility See specific details

Have a fun !