Read PG in data warehouse in one article_ stat

Abstract ：GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory . This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

This article is shared from Huawei cloud community 《 Article to read pgstat》, author ： leapdb .

GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory .

This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

pg_stat_get_tuples_inserted   -- Table accumulation insert Number of pieces 
pg_stat_get_tuples_updated    -- Table accumulation update Number of pieces 
pg_stat_get_tuples_deleted    -- Table accumulation delete Number of pieces 
pg_stat_get_tuples_changed    -- Table since last analyze since , Number of changes 
pg_stat_get_last_analyze_time -- Query the last analyze Time 

therefore , Based on shared memory " Table since last analyze Number of entries modified since " Whether a certain threshold is exceeded , You can decide whether you need to do analyze 了 .

1. How to record statistics in a transaction

Every session There is one PgStat_TableStatus Memory structure , Each table accessed by the current transaction occupies one .

It contains the statistics of each level of sub transaction , With the structure PgStat_TableXactStatus describe . Various information about the addition, deletion, modification and query of the current sub transaction is recorded here .

pg_stat_get_xact_numscans -- The current transaction is on the current table , Number of times to start sequential scanning 
pg_stat_get_xact_tuples_returned -- The current transaction is on the current table , Number of visible tuples captured by sequential scanning .
pg_stat_get_xact_tuples_fetched -- The current transaction is on the current table , The number of visible tuples captured .
pg_stat_get_xact_tuples_inserted -- The current transaction is on the current table , Insert the number of 
pg_stat_get_xact_tuples_deleted -- The current transaction is on the current table , Delete the number 
pg_stat_get_xact_tuples_updated -- The current transaction is on the current table , Number of updates 
pg_stat_get_xact_tuples_hot_updated -- The current transaction is on the current table ,HOT Number of updates 
pg_stat_get_xact_blocks_fetched -- The current transaction is on the current table , Number of pages selected 
pg_stat_get_xact_blocks_hit -- The current transaction is on the current table , Number of pages scanned 
pg_stat_get_xact_function_calls -- The current transaction is on the current function , Call the number 
pg_stat_get_xact_function_total_time -- The current transaction is on the current function , Total execution time of all calls 
pg_stat_get_xact_function_self_time -- The current transaction is on the current function , The execution time of the last call 
pg_stat_get_xact_partition_tuples_inserted -- The current transaction is on the current table partition , Insert the number of 
pg_stat_get_xact_partition_tuples_deleted -- The current transaction is on the current table partition , Delete the number 
pg_stat_get_xact_partition_tuples_updated -- The current transaction is on the current table partition , Number of updates 
pg_stat_get_xact_partition_tuples_hot_updated -- The current transaction is on the current table partition ,HOT Number of updates 

When a transaction is committed, the current table , Summary of information in all sub transactions , And dump to pgstat in .

2. How to record statistics outside a transaction

When a transaction is committed, the execution information in the transaction is passed through UDP A message is sent to PgCollector Background thread , It records to the unified memory , And dump it to the physical file when the cluster stops normally pg_stat_tmp/pgstat.stat in .

• Instance level information ,PgStat_GlobalStats
• Library level information ,PgStat_StatDBEntry
• Table level information , Its memory structure PgStat_StatTabEntry

-- Instance level information 
pg_stat_get_bgwriter_timed_checkpoints -- The number of times the background writer started the scheduled checkpoint 
pg_stat_get_bgwriter_requested_checkpoints -- The number of times the background writer has opened checkpoints based on back-end requests 
pg_stat_get_checkpoint_write_time -- The last time checkpoint Time to write pages 
pg_stat_get_checkpoint_sync_time -- The last time checkpoint Time to synchronize pages 
pg_stat_get_bgwriter_buf_written_checkpoints -- Number of buffers written by the background writer during checkpoint .
pg_stat_get_bgwriter_buf_written_clean -- Clean up dirty blocks for daily use , Number of buffers written by the background writer process .
pg_stat_get_bgwriter_maxwritten_clean -- The time when the background writer process stopped cleaning up the scan , Because more buffers have been written （ comparison bgwriter_lru_maxpages Number of buffers declared by parameter ）.
pg_stat_get_buf_written_backend -- Number of buffers written by the backend process , Because they need to allocate a new buffer .
pg_stat_get_buf_fsync_backend -- Background process execution fsync The number of times 
pg_stat_get_buf_alloc -- Total number of buffers allocated .
pg_stat_get_bgwriter_stat_reset_time -- backstage bgwriter Thread reset time 

-- At present CN Recorded library level information 
pg_stat_get_db_xact_commit -- The number of committed transactions in the specified database on the current instance .
pg_stat_get_db_xact_rollback -- The number of rollback transactions in the specified database on the current instance .
pg_stat_get_db_blocks_fetched -- The number of disk block fetching requests in the specified database on the current instance .
pg_stat_get_db_blocks_hit -- The number of requested disk blocks found by the specified database in the buffer on the current instance .
pg_stat_get_db_tuples_returned -- The number of primitives returned by the specified database on the current instance .
pg_stat_get_db_tuples_fetched -- The number of tuples read from the specified database on the current instance .
pg_stat_get_db_tuples_inserted -- The number of tuples inserted in the specified database on the current instance .
pg_stat_get_db_tuples_updated -- The number of tuples updated in the specified database on the current instance .
pg_stat_get_db_tuples_deleted -- The number of tuples deleted from the specified database on the current instance .
pg_stat_get_db_conflict_all -- The number of times conflict recovery occurred in the specified database on the current instance .
pg_stat_get_db_conflict_lock -- All in the cluster CN and DN Total number of database lock conflicts specified on . stay DN Execute this function on , Returns the number of lock conflicts in the specified database on the current instance .
pg_stat_get_db_deadlocks -- The number of deadlocks in the specified database on the current instance .
pg_stat_get_db_temp_files -- The number of temporary files created in the specified database on the current instance .
pg_stat_get_db_temp_bytes -- The number of bytes to create temporary files in the specified database on the current instance .
pg_stat_get_db_blk_read_time -- The time taken to read data blocks in the specified database on the current instance .
pg_stat_get_db_blk_write_time -- The time taken to write data blocks in the specified database on the current instance .

-- Library level information recorded by the entire cluster 
pg_stat_get_db_total_xact_commit -- stay CN Execute this function on , Return to all... In the cluster CN The total number of committed transactions in the specified database on . stay DN Execute this function on , Returns the number of committed transactions in the specified database on the current instance .
pg_stat_get_db_total_xact_rollback -- stay CN Execute this function on , Return to all... In the cluster CN The total number of rollback transactions in the specified database on . stay DN Execute this function on , Returns the number of rollback transactions in the specified database on the current instance .
pg_stat_get_db_total_blocks_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of disk block fetch requests in the specified database on . stay DN Execute this function on , Returns the number of disk block fetching requests in the specified database on the current instance .
pg_stat_get_db_total_blocks_hit -- stay CN Execute this function on , Return to all... In the cluster DN The total number of requested disk blocks found in the buffer by the specified database on . stay DN Execute this function on , Returns the number of requested disk blocks found in the buffer by the specified database on the current instance .
pg_stat_get_db_total_tuples_returned -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples returned by the specified database on . stay DN Execute this function on , Returns the number of tuples returned by the specified database on the current instance .
pg_stat_get_db_total_tuples_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples read from the specified database on . stay DN Execute this function on , Returns the number of tuples read from the specified database on the current instance .
pg_stat_get_db_total_tuples_inserted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples inserted on the specified database . stay DN Execute this function on , Returns the number of tuples inserted in the specified database on the current instance .
pg_stat_get_db_total_tuples_updated -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples updated on the specified database . stay DN Execute this function on , Returns the number of tuples updated by the specified database on the current instance .
pg_stat_get_db_total_tuples_deleted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples deleted from the specified database on . stay DN Execute this function on , Returns the number of tuples deleted from the specified database on the current instance .
pg_stat_get_db_total_conflict_all -- stay CN Execute this function on , Return to all... In the cluster CN and DN Total number of conflict recoveries for the specified database on . stay DN Execute this function on , Returns the number of conflict recoveries in the specified database on the current instance .
pg_stat_get_db_total_temp_files -- stay CN Execute this function on , Return to all... In the cluster DN The total number of temporary files created in the specified database on . stay DN Execute this function on , Returns the number of temporary files created in the specified database on the current instance .
pg_stat_get_db_total_temp_bytes -- stay CN Execute this function on , Return to all... In the cluster DN Total number of bytes to create temporary files in the specified database on . stay DN Execute this function on , Returns the number of bytes to create a temporary file in the specified database on the current instance .
pg_stat_get_db_total_deadlocks -- All in the cluster CN and DN Total number of database deadlocks specified on . stay DN Execute this function on , Returns the number of deadlocks in the specified database on the current instance .
pg_stat_get_db_total_blk_read_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent reading data blocks in the specified database on . stay DN Execute this function on , Returns the time spent reading data blocks in the specified database on the current instance .
pg_stat_get_db_total_blk_write_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent writing data blocks in the specified database on . stay DN Execute this function on , Returns the time spent writing data blocks in the specified database on the current instance .

-- Table level information 
pg_stat_get_numscans -- On the current table , Number of times to start sequential scanning 
pg_stat_get_tuples_returned -- On the current table , Number of visible tuples captured by sequential scanning .
pg_stat_get_tuples_fetched -- On the current table , The number of visible tuples captured .
pg_stat_get_tuples_inserted -- On the current table , Insert the number of 
pg_stat_get_tuples_deleted -- On the current table , Delete the number 
pg_stat_get_tuples_updated -- On the current table , Number of updates 
pg_stat_get_tuples_hot_updated -- On the current table ,HOT Number of updates 
pg_stat_get_blocks_fetched -- On the current table , Number of pages selected 
pg_stat_get_blocks_hit -- On the current table , Number of pages scanned 
pg_stat_get_function_calls -- On the current function , Call the number 
pg_stat_get_function_total_time -- On the current function , Total execution time of all calls 
pg_stat_get_function_self_time -- On the current function , The execution time of the last call 
pg_stat_get_partition_tuples_inserted -- On the current table partition , Insert the number of 
pg_stat_get_partition_tuples_deleted -- On the current table partition , Delete the number 
pg_stat_get_partition_tuples_updated -- On the current table partition , Number of updates 
pg_stat_get_partition_tuples_hot_updated -- On the current table partition ,HOT Number of updates 
pg_stat_get_tuples_changed -- On the current table since the last analyze Historical cumulative change of 
pg_stat_get_partition_tuples_changed -- On the current table partition since the last analyze Historical cumulative change of 
pg_stat_get_partition_live_tuples -- The number of tuples visible on the current table partition 
pg_stat_get_partition_dead_tuples -- The number of tuples deleted on the current table partition 
pg_stat_get_live_tuples -- The number of tuples visible on the current table 
pg_stat_get_last_vacuum_time -- The last time on the current table vacuum Time for 
pg_stat_get_last_data_changed_time -- Last data modification time on the current table 
pg_stat_get_last_autovacuum_time -- The last time on the current table autovacuum Time 
pg_stat_get_last_autoanalyze_time -- The last time on the current table autoanalyze Time 
pg_stat_get_last_analyze_time -- The last manual operation on the current table analyze Time 

--local Indicates the information of the current node 
pg_stat_get_local_tuples_updated
pg_stat_get_local_tuples_inserted
pg_stat_get_local_tuples_hot_updated
pg_stat_get_local_tuples_deleted
pg_stat_get_local_tuples_changed
pg_stat_get_local_live_tuples
pg_stat_get_local_last_autovacuum_time
pg_stat_get_local_last_autoanalyze_time
pg_stat_get_local_dead_tuples
pg_stat_get_local_autovacuum_count
pg_stat_get_local_autoanalyze_count
pg_stat_get_local_analyze_status

The function of information ,PgStat_StatFuncEntry

pg_stat_get_function_calls -- The number of times the function has been called .
pg_stat_get_function_total_time -- Total wall clock time spent by this function , In microseconds . Include the time spent on this function call .
pg_stat_get_function_self_time -- Only time spent on this function in the current transaction . It does not include the time spent calling functions .

3. Single node statistics

Every CN and DN node , Record their own statistics . You can use the above functions to view .

in addition ,CN In addition, deletion and modification SQL At the end of execution , Will all DN The number of returned pieces of information is recorded to CN Their own pgstat In structure , Constitute the overall statistical information .

4. Cluster global statistics

each CN Each of us is defending our own pgstat Information , Therefore, if you want to know the global , Also need to report to all CN Query and summarize .

5. The lifecycle of statistics

pgstat The statistics in are runtime information , This information is outside the transaction , Data consistency is not strictly guaranteed .

When the database stops abnormally, all data will be cleared , A normal stop preserves the data .

6. Maintenance of statistical information

This information is in execution SQL Is automatically recorded , No manual maintenance is required .

Click to follow , The first time to learn about Huawei's new cloud technology ~