Read PG in data warehouse in one article_ stat

Abstract ：GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory . This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

This article is shared from Huawei cloud community 《 Article to read pgstat》, author ： leapdb .

GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory .

This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

pg_stat_get_tuples_inserted   -- Table accumulation insert Number of pieces pg_stat_get_tuples_updated    -- Table accumulation update Number of pieces pg_stat_get_tuples_deleted    -- Table accumulation delete Number of pieces pg_stat_get_tuples_changed    -- Table since last analyze since , Number of changes pg_stat_get_last_analyze_time -- Query the last analyze Time 

therefore , Based on shared memory " Table since last analyze Number of entries modified since " Whether a certain threshold is exceeded , You can decide whether you need to do analyze 了 .

1. How to record statistics in a transaction

Every session There is one PgStat_TableStatus Memory structure , Each table accessed by the current transaction occupies one .

It contains the statistics of each level of sub transaction , With the structure PgStat_TableXactStatus describe . Various information about the addition, deletion, modification and query of the current sub transaction is recorded here .

pg_stat_get_xact_numscans -- The current transaction is on the current table , Number of times to start sequential scanning pg_stat_get_xact_tuples_returned -- The current transaction is on the current table , Number of visible tuples captured by sequential scanning .pg_stat_get_xact_tuples_fetched -- The current transaction is on the current table , The number of visible tuples captured .pg_stat_get_xact_tuples_inserted -- The current transaction is on the current table , Insert the number of pg_stat_get_xact_tuples_deleted -- The current transaction is on the current table , Delete the number pg_stat_get_xact_tuples_updated -- The current transaction is on the current table , Number of updates pg_stat_get_xact_tuples_hot_updated -- The current transaction is on the current table ,HOT Number of updates pg_stat_get_xact_blocks_fetched -- The current transaction is on the current table , Number of pages selected pg_stat_get_xact_blocks_hit -- The current transaction is on the current table , Number of pages scanned pg_stat_get_xact_function_calls -- The current transaction is on the current function , Call the number pg_stat_get_xact_function_total_time -- The current transaction is on the current function , Total execution time of all calls pg_stat_get_xact_function_self_time -- The current transaction is on the current function , The execution time of the last call pg_stat_get_xact_partition_tuples_inserted -- The current transaction is on the current table partition , Insert the number of pg_stat_get_xact_partition_tuples_deleted -- The current transaction is on the current table partition , Delete the number pg_stat_get_xact_partition_tuples_updated -- The current transaction is on the current table partition , Number of updates pg_stat_get_xact_partition_tuples_hot_updated -- The current transaction is on the current table partition ,HOT Number of updates 

When a transaction is committed, the current table , Summary of information in all sub transactions , And dump to pgstat in .

2. How to record statistics outside a transaction

When a transaction is committed, the execution information in the transaction is passed through UDP A message is sent to PgCollector Background thread , It records to the unified memory , And dump it to the physical file when the cluster stops normally pg_stat_tmp/pgstat.stat in .

• Instance level information ,PgStat_GlobalStats
• Library level information ,PgStat_StatDBEntry
• Table level information , Its memory structure PgStat_StatTabEntry

-- Instance level information pg_stat_get_bgwriter_timed_checkpoints -- The number of times the background writer started the scheduled checkpoint pg_stat_get_bgwriter_requested_checkpoints -- The number of times the background writer has opened checkpoints based on back-end requests pg_stat_get_checkpoint_write_time -- The last time checkpoint Time to write pages pg_stat_get_checkpoint_sync_time -- The last time checkpoint Time to synchronize pages pg_stat_get_bgwriter_buf_written_checkpoints -- Number of buffers written by the background writer during checkpoint .pg_stat_get_bgwriter_buf_written_clean -- Clean up dirty blocks for daily use , Number of buffers written by the background writer process .pg_stat_get_bgwriter_maxwritten_clean -- The time when the background writer process stopped cleaning up the scan , Because more buffers have been written （ comparison bgwriter_lru_maxpages Number of buffers declared by parameter ）.pg_stat_get_buf_written_backend -- Number of buffers written by the backend process , Because they need to allocate a new buffer .pg_stat_get_buf_fsync_backend -- Background process execution fsync The number of times pg_stat_get_buf_alloc -- Total number of buffers allocated .pg_stat_get_bgwriter_stat_reset_time -- backstage bgwriter Thread reset time -- At present CN Recorded library level information pg_stat_get_db_xact_commit -- The number of committed transactions in the specified database on the current instance .pg_stat_get_db_xact_rollback -- The number of rollback transactions in the specified database on the current instance .pg_stat_get_db_blocks_fetched -- The number of disk block fetching requests in the specified database on the current instance .pg_stat_get_db_blocks_hit -- The number of requested disk blocks found by the specified database in the buffer on the current instance .pg_stat_get_db_tuples_returned -- The number of primitives returned by the specified database on the current instance .pg_stat_get_db_tuples_fetched -- The number of tuples read from the specified database on the current instance .pg_stat_get_db_tuples_inserted -- The number of tuples inserted in the specified database on the current instance .pg_stat_get_db_tuples_updated -- The number of tuples updated in the specified database on the current instance .pg_stat_get_db_tuples_deleted -- The number of tuples deleted from the specified database on the current instance .pg_stat_get_db_conflict_all -- The number of times conflict recovery occurred in the specified database on the current instance .pg_stat_get_db_conflict_lock -- All in the cluster CN and DN Total number of database lock conflicts specified on . stay DN Execute this function on , Returns the number of lock conflicts in the specified database on the current instance .pg_stat_get_db_deadlocks -- The number of deadlocks in the specified database on the current instance .pg_stat_get_db_temp_files -- The number of temporary files created in the specified database on the current instance .pg_stat_get_db_temp_bytes -- The number of bytes to create temporary files in the specified database on the current instance .pg_stat_get_db_blk_read_time -- The time taken to read data blocks in the specified database on the current instance .pg_stat_get_db_blk_write_time -- The time taken to write data blocks in the specified database on the current instance .-- Library level information recorded by the entire cluster pg_stat_get_db_total_xact_commit -- stay CN Execute this function on , Return to all... In the cluster CN The total number of committed transactions in the specified database on . stay DN Execute this function on , Returns the number of committed transactions in the specified database on the current instance .pg_stat_get_db_total_xact_rollback -- stay CN Execute this function on , Return to all... In the cluster CN The total number of rollback transactions in the specified database on . stay DN Execute this function on , Returns the number of rollback transactions in the specified database on the current instance .pg_stat_get_db_total_blocks_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of disk block fetch requests in the specified database on . stay DN Execute this function on , Returns the number of disk block fetching requests in the specified database on the current instance .pg_stat_get_db_total_blocks_hit -- stay CN Execute this function on , Return to all... In the cluster DN The total number of requested disk blocks found in the buffer by the specified database on . stay DN Execute this function on , Returns the number of requested disk blocks found in the buffer by the specified database on the current instance .pg_stat_get_db_total_tuples_returned -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples returned by the specified database on . stay DN Execute this function on , Returns the number of tuples returned by the specified database on the current instance .pg_stat_get_db_total_tuples_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples read from the specified database on . stay DN Execute this function on , Returns the number of tuples read from the specified database on the current instance .pg_stat_get_db_total_tuples_inserted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples inserted on the specified database . stay DN Execute this function on , Returns the number of tuples inserted in the specified database on the current instance .pg_stat_get_db_total_tuples_updated -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples updated on the specified database . stay DN Execute this function on , Returns the number of tuples updated by the specified database on the current instance .pg_stat_get_db_total_tuples_deleted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples deleted from the specified database on . stay DN Execute this function on , Returns the number of tuples deleted from the specified database on the current instance .pg_stat_get_db_total_conflict_all -- stay CN Execute this function on , Return to all... In the cluster CN and DN Total number of conflict recoveries for the specified database on . stay DN Execute this function on , Returns the number of conflict recoveries in the specified database on the current instance .pg_stat_get_db_total_temp_files -- stay CN Execute this function on , Return to all... In the cluster DN The total number of temporary files created in the specified database on . stay DN Execute this function on , Returns the number of temporary files created in the specified database on the current instance .pg_stat_get_db_total_temp_bytes -- stay CN Execute this function on , Return to all... In the cluster DN Total number of bytes to create temporary files in the specified database on . stay DN Execute this function on , Returns the number of bytes to create a temporary file in the specified database on the current instance .pg_stat_get_db_total_deadlocks -- All in the cluster CN and DN Total number of database deadlocks specified on . stay DN Execute this function on , Returns the number of deadlocks in the specified database on the current instance .pg_stat_get_db_total_blk_read_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent reading data blocks in the specified database on . stay DN Execute this function on , Returns the time spent reading data blocks in the specified database on the current instance .pg_stat_get_db_total_blk_write_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent writing data blocks in the specified database on . stay DN Execute this function on , Returns the time spent writing data blocks in the specified database on the current instance .-- Table level information pg_stat_get_numscans -- On the current table , Number of times to start sequential scanning pg_stat_get_tuples_returned -- On the current table , Number of visible tuples captured by sequential scanning .pg_stat_get_tuples_fetched -- On the current table , The number of visible tuples captured .pg_stat_get_tuples_inserted -- On the current table , Insert the number of pg_stat_get_tuples_deleted -- On the current table , Delete the number pg_stat_get_tuples_updated -- On the current table , Number of updates pg_stat_get_tuples_hot_updated -- On the current table ,HOT Number of updates pg_stat_get_blocks_fetched -- On the current table , Number of pages selected pg_stat_get_blocks_hit -- On the current table , Number of pages scanned pg_stat_get_function_calls -- On the current function , Call the number pg_stat_get_function_total_time -- On the current function , Total execution time of all calls pg_stat_get_function_self_time -- On the current function , The execution time of the last call pg_stat_get_partition_tuples_inserted -- On the current table partition , Insert the number of pg_stat_get_partition_tuples_deleted -- On the current table partition , Delete the number pg_stat_get_partition_tuples_updated -- On the current table partition , Number of updates pg_stat_get_partition_tuples_hot_updated -- On the current table partition ,HOT Number of updates pg_stat_get_tuples_changed -- On the current table since the last analyze Historical cumulative change of pg_stat_get_partition_tuples_changed -- On the current table partition since the last analyze Historical cumulative change of pg_stat_get_partition_live_tuples -- The number of tuples visible on the current table partition pg_stat_get_partition_dead_tuples -- The number of tuples deleted on the current table partition pg_stat_get_live_tuples -- The number of tuples visible on the current table pg_stat_get_last_vacuum_time -- The last time on the current table vacuum Time for pg_stat_get_last_data_changed_time -- Last data modification time on the current table pg_stat_get_last_autovacuum_time -- The last time on the current table autovacuum Time pg_stat_get_last_autoanalyze_time -- The last time on the current table autoanalyze Time pg_stat_get_last_analyze_time -- The last manual operation on the current table analyze Time --local Indicates the information of the current node pg_stat_get_local_tuples_updatedpg_stat_get_local_tuples_insertedpg_stat_get_local_tuples_hot_updatedpg_stat_get_local_tuples_deletedpg_stat_get_local_tuples_changedpg_stat_get_local_live_tuplespg_stat_get_local_last_autovacuum_timepg_stat_get_local_last_autoanalyze_timepg_stat_get_local_dead_tuplespg_stat_get_local_autovacuum_countpg_stat_get_local_autoanalyze_countpg_stat_get_local_analyze_status

The function of information ,PgStat_StatFuncEntry

pg_stat_get_function_calls -- The number of times the function has been called .pg_stat_get_function_total_time -- Total wall clock time spent by this function , In microseconds . Include the time spent on this function call .pg_stat_get_function_self_time -- Only time spent on this function in the current transaction . It does not include the time spent calling functions .

3. Single node statistics

Every CN and DN node , Record their own statistics . You can use the above functions to view .

in addition ,CN In addition, deletion and modification SQL At the end of execution , Will all DN The number of returned pieces of information is recorded to CN Their own pgstat In structure , Constitute the overall statistical information .

4. Cluster global statistics

each CN Each of us is defending our own pgstat Information , Therefore, if you want to know the global , Also need to report to all CN Query and summarize .

5. The lifecycle of statistics

pgstat The statistics in are runtime information , This information is outside the transaction , Data consistency is not strictly guaranteed .

When the database stops abnormally, all data will be cleared , A normal stop preserves the data .

6. Maintenance of statistical information

This information is in execution SQL Is automatically recorded , No manual maintenance is required .

 

Click to follow , The first time to learn about Huawei's new cloud technology ~