Abstract ：GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory . This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

This article is shared from Huawei cloud community 《 Article to read pgstat》, author ： leapdb .

GaussDB(DWS) stay SQL In the process of execution , It will record the addition, deletion, modification and query of relevant runtime Statistics , After the transaction is committed or rolled back, it is recorded in the shared memory .

This information can be obtained through “pg_stat_all_tables View ” Inquire about , You can also query through the following functions .

pg_stat_get_tuples_inserted   -- Table accumulation insert Number of pieces 

pg_stat_get_tuples_updated    -- Table accumulation update Number of pieces 

pg_stat_get_tuples_deleted    -- Table accumulation delete Number of pieces 

pg_stat_get_tuples_changed    -- Table since last analyze since , Number of changes 

pg_stat_get_last_analyze_time -- Query the last analyze Time

therefore , Based on shared memory " Table since last analyze Number of entries modified since " Whether a certain threshold is exceeded , You can decide whether you need to do analyze 了 .

1. How to record statistics in a transaction

Every session There is one PgStat_TableStatus Memory structure , Each table accessed by the current transaction occupies one .

It contains the statistics of each level of sub transaction , With the structure PgStat_TableXactStatus describe . Various information about the addition, deletion, modification and query of the current sub transaction is recorded here .

pg_stat_get_xact_numscans -- The current transaction is on the current table , Number of times to start sequential scanning 

pg_stat_get_xact_tuples_returned -- The current transaction is on the current table , Number of visible tuples captured by sequential scanning .

pg_stat_get_xact_tuples_fetched -- The current transaction is on the current table , The number of visible tuples captured .

pg_stat_get_xact_tuples_inserted -- The current transaction is on the current table , Insert the number of 

pg_stat_get_xact_tuples_deleted -- The current transaction is on the current table , Delete the number 

pg_stat_get_xact_tuples_updated -- The current transaction is on the current table , Number of updates 

pg_stat_get_xact_tuples_hot_updated -- The current transaction is on the current table ,HOT Number of updates 

pg_stat_get_xact_blocks_fetched -- The current transaction is on the current table , Number of pages selected 

pg_stat_get_xact_blocks_hit -- The current transaction is on the current table , Number of pages scanned 

pg_stat_get_xact_function_calls -- The current transaction is on the current function , Call the number 

pg_stat_get_xact_function_total_time -- The current transaction is on the current function , Total execution time of all calls 

pg_stat_get_xact_function_self_time -- The current transaction is on the current function , The execution time of the last call 

pg_stat_get_xact_partition_tuples_inserted -- The current transaction is on the current table partition , Insert the number of 

pg_stat_get_xact_partition_tuples_deleted -- The current transaction is on the current table partition , Delete the number 

pg_stat_get_xact_partition_tuples_updated -- The current transaction is on the current table partition , Number of updates 

pg_stat_get_xact_partition_tuples_hot_updated -- The current transaction is on the current table partition ,HOT Number of updates

When a transaction is committed, the current table , Summary of information in all sub transactions , And dump to pgstat in .

2. How to record statistics outside a transaction

When a transaction is committed, the execution information in the transaction is passed through UDP A message is sent to PgCollector Background thread , It records to the unified memory , And dump it to the physical file when the cluster stops normally pg_stat_tmp/pgstat.stat in .

  • Instance level information ,PgStat_GlobalStats
  • Library level information ,PgStat_StatDBEntry
  • Table level information , Its memory structure PgStat_StatTabEntry
-- Instance level information 

pg_stat_get_bgwriter_timed_checkpoints -- The number of times the background writer started the scheduled checkpoint 

pg_stat_get_bgwriter_requested_checkpoints -- The number of times the background writer has opened checkpoints based on back-end requests 

pg_stat_get_checkpoint_write_time -- The last time checkpoint Time to write pages 

pg_stat_get_checkpoint_sync_time -- The last time checkpoint Time to synchronize pages 

pg_stat_get_bgwriter_buf_written_checkpoints -- Number of buffers written by the background writer during checkpoint .

pg_stat_get_bgwriter_buf_written_clean -- Clean up dirty blocks for daily use , Number of buffers written by the background writer process .

pg_stat_get_bgwriter_maxwritten_clean -- The time when the background writer process stopped cleaning up the scan , Because more buffers have been written （ comparison bgwriter_lru_maxpages Number of buffers declared by parameter ）.

pg_stat_get_buf_written_backend -- Number of buffers written by the backend process , Because they need to allocate a new buffer .

pg_stat_get_buf_fsync_backend -- Background process execution fsync The number of times 

pg_stat_get_buf_alloc -- Total number of buffers allocated .

pg_stat_get_bgwriter_stat_reset_time -- backstage bgwriter Thread reset time 

-- At present CN Recorded library level information 

pg_stat_get_db_xact_commit -- The number of committed transactions in the specified database on the current instance .

pg_stat_get_db_xact_rollback -- The number of rollback transactions in the specified database on the current instance .

pg_stat_get_db_blocks_fetched -- The number of disk block fetching requests in the specified database on the current instance .

pg_stat_get_db_blocks_hit -- The number of requested disk blocks found by the specified database in the buffer on the current instance .

pg_stat_get_db_tuples_returned -- The number of primitives returned by the specified database on the current instance .

pg_stat_get_db_tuples_fetched -- The number of tuples read from the specified database on the current instance .

pg_stat_get_db_tuples_inserted -- The number of tuples inserted in the specified database on the current instance .

pg_stat_get_db_tuples_updated -- The number of tuples updated in the specified database on the current instance .

pg_stat_get_db_tuples_deleted -- The number of tuples deleted from the specified database on the current instance .

pg_stat_get_db_conflict_all -- The number of times conflict recovery occurred in the specified database on the current instance .

pg_stat_get_db_conflict_lock -- All in the cluster CN and DN Total number of database lock conflicts specified on . stay DN Execute this function on , Returns the number of lock conflicts in the specified database on the current instance .

pg_stat_get_db_deadlocks -- The number of deadlocks in the specified database on the current instance .

pg_stat_get_db_temp_files -- The number of temporary files created in the specified database on the current instance .

pg_stat_get_db_temp_bytes -- The number of bytes to create temporary files in the specified database on the current instance .

pg_stat_get_db_blk_read_time -- The time taken to read data blocks in the specified database on the current instance .

pg_stat_get_db_blk_write_time -- The time taken to write data blocks in the specified database on the current instance .

-- Library level information recorded by the entire cluster 

pg_stat_get_db_total_xact_commit -- stay CN Execute this function on , Return to all... In the cluster CN The total number of committed transactions in the specified database on . stay DN Execute this function on , Returns the number of committed transactions in the specified database on the current instance .

pg_stat_get_db_total_xact_rollback -- stay CN Execute this function on , Return to all... In the cluster CN The total number of rollback transactions in the specified database on . stay DN Execute this function on , Returns the number of rollback transactions in the specified database on the current instance .

pg_stat_get_db_total_blocks_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of disk block fetch requests in the specified database on . stay DN Execute this function on , Returns the number of disk block fetching requests in the specified database on the current instance .

pg_stat_get_db_total_blocks_hit -- stay CN Execute this function on , Return to all... In the cluster DN The total number of requested disk blocks found in the buffer by the specified database on . stay DN Execute this function on , Returns the number of requested disk blocks found in the buffer by the specified database on the current instance .

pg_stat_get_db_total_tuples_returned -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples returned by the specified database on . stay DN Execute this function on , Returns the number of tuples returned by the specified database on the current instance .

pg_stat_get_db_total_tuples_fetched -- stay CN Execute this function on , Return to all... In the cluster DN The total number of tuples read from the specified database on . stay DN Execute this function on , Returns the number of tuples read from the specified database on the current instance .

pg_stat_get_db_total_tuples_inserted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples inserted on the specified database . stay DN Execute this function on , Returns the number of tuples inserted in the specified database on the current instance .

pg_stat_get_db_total_tuples_updated -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples updated on the specified database . stay DN Execute this function on , Returns the number of tuples updated by the specified database on the current instance .

pg_stat_get_db_total_tuples_deleted -- stay CN Execute this function on , Return to all... In the cluster DN Total number of tuples deleted from the specified database on . stay DN Execute this function on , Returns the number of tuples deleted from the specified database on the current instance .

pg_stat_get_db_total_conflict_all -- stay CN Execute this function on , Return to all... In the cluster CN and DN Total number of conflict recoveries for the specified database on . stay DN Execute this function on , Returns the number of conflict recoveries in the specified database on the current instance .

pg_stat_get_db_total_temp_files -- stay CN Execute this function on , Return to all... In the cluster DN The total number of temporary files created in the specified database on . stay DN Execute this function on , Returns the number of temporary files created in the specified database on the current instance .

pg_stat_get_db_total_temp_bytes -- stay CN Execute this function on , Return to all... In the cluster DN Total number of bytes to create temporary files in the specified database on . stay DN Execute this function on , Returns the number of bytes to create a temporary file in the specified database on the current instance .

pg_stat_get_db_total_deadlocks -- All in the cluster CN and DN Total number of database deadlocks specified on . stay DN Execute this function on , Returns the number of deadlocks in the specified database on the current instance .

pg_stat_get_db_total_blk_read_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent reading data blocks in the specified database on . stay DN Execute this function on , Returns the time spent reading data blocks in the specified database on the current instance .

pg_stat_get_db_total_blk_write_time -- stay CN Execute this function on , Return to all... In the cluster DN Total time spent writing data blocks in the specified database on . stay DN Execute this function on , Returns the time spent writing data blocks in the specified database on the current instance .

-- Table level information 

pg_stat_get_numscans -- On the current table , Number of times to start sequential scanning 

pg_stat_get_tuples_returned -- On the current table , Number of visible tuples captured by sequential scanning .

pg_stat_get_tuples_fetched -- On the current table , The number of visible tuples captured .

pg_stat_get_tuples_inserted -- On the current table , Insert the number of 

pg_stat_get_tuples_deleted -- On the current table , Delete the number 

pg_stat_get_tuples_updated -- On the current table , Number of updates 

pg_stat_get_tuples_hot_updated -- On the current table ,HOT Number of updates 

pg_stat_get_blocks_fetched -- On the current table , Number of pages selected 

pg_stat_get_blocks_hit -- On the current table , Number of pages scanned 

pg_stat_get_function_calls -- On the current function , Call the number 

pg_stat_get_function_total_time -- On the current function , Total execution time of all calls 

pg_stat_get_function_self_time -- On the current function , The execution time of the last call 

pg_stat_get_partition_tuples_inserted -- On the current table partition , Insert the number of 

pg_stat_get_partition_tuples_deleted -- On the current table partition , Delete the number 

pg_stat_get_partition_tuples_updated -- On the current table partition , Number of updates 

pg_stat_get_partition_tuples_hot_updated -- On the current table partition ,HOT Number of updates 

pg_stat_get_tuples_changed -- On the current table since the last analyze Historical cumulative change of 

pg_stat_get_partition_tuples_changed -- On the current table partition since the last analyze Historical cumulative change of 

pg_stat_get_partition_live_tuples -- The number of tuples visible on the current table partition 

pg_stat_get_partition_dead_tuples -- The number of tuples deleted on the current table partition 

pg_stat_get_live_tuples -- The number of tuples visible on the current table 

pg_stat_get_last_vacuum_time -- The last time on the current table vacuum Time for 

pg_stat_get_last_data_changed_time -- Last data modification time on the current table 

pg_stat_get_last_autovacuum_time -- The last time on the current table autovacuum Time 

pg_stat_get_last_autoanalyze_time -- The last time on the current table autoanalyze Time 

pg_stat_get_last_analyze_time -- The last manual operation on the current table analyze Time 

--local Indicates the information of the current node 

pg_stat_get_local_tuples_updated

pg_stat_get_local_tuples_inserted

pg_stat_get_local_tuples_hot_updated

pg_stat_get_local_tuples_deleted

pg_stat_get_local_tuples_changed

pg_stat_get_local_live_tuples

pg_stat_get_local_last_autovacuum_time

pg_stat_get_local_last_autoanalyze_time

pg_stat_get_local_dead_tuples

pg_stat_get_local_autovacuum_count

pg_stat_get_local_autoanalyze_count

pg_stat_get_local_analyze_status

The function of information ,PgStat_StatFuncEntry

pg_stat_get_function_calls -- The number of times the function has been called .

pg_stat_get_function_total_time -- Total wall clock time spent by this function , In microseconds . Include the time spent on this function call .

pg_stat_get_function_self_time -- Only time spent on this function in the current transaction . It does not include the time spent calling functions .

3. Single node statistics

Every CN and DN node , Record their own statistics . You can use the above functions to view .

in addition ,CN In addition, deletion and modification SQL At the end of execution , Will all DN The number of returned pieces of information is recorded to CN Their own pgstat In structure , Constitute the overall statistical information .

4. Cluster global statistics

each CN Each of us is defending our own pgstat Information , Therefore, if you want to know the global , Also need to report to all CN Query and summarize .

5. The lifecycle of statistics

pgstat The statistics in are runtime information , This information is outside the transaction , Data consistency is not strictly guaranteed .

When the database stops abnormally, all data will be cleared , A normal stop preserves the data .

6. Maintenance of statistical information

This information is in execution SQL Is automatically recorded , No manual maintenance is required .

Click to follow , The first time to learn about Huawei's new cloud technology ~

Read and understand... In the data warehouse in one article pg_stat More articles about

  1. In this article, I will read and understand... In neural network training Batch Size,Epoch,Iteration

    In this article, I will read and understand... In neural network training Batch Size,Epoch,Iteration As a few nouns that can't be avoided in all kinds of neural network training , This article will comprehensively analyze their meaning and relationship . 1. Batch Size paraphrase : Batch size , ...

  2. Read and understand high performance network programming I/O Model

    1. Preface With the development of the Internet , In the face of massive users and high concurrency business , The traditional blocking server-side architecture model has been powerless . this paper ( And the next part < High performance network programming ( 6、 ... and ): Read the thread model in high performance network programming >) It aims to provide you with useful ...

  3. Big data ： Article to read @ Data warehouse (PPT Text version )

    Big data : Article to read @ Data warehouse 1 Network Vocabulary summary 1.1 Data center Data in the middle is aggregation and governance of cross domain data , Encapsulate data abstraction into services , Provide the front desk with a logical concept of business value . The data center is a set of sustainable " Let the data of enterprises use ...

  4. Article to read HTTP/2 And HTTP/3 characteristic

    Abstract : Study HTTP/2 And HTTP/3. Preface HTTP/2 Compared with HTTP/1, It can be said that the performance of the web page has been greatly improved , Only upgrading to this protocol can reduce a lot of previous performance optimization work , Of course, compatibility issues and how to ...

  5. from HTTP/0.9 To HTTP/2： Article to read HTTP The historical evolution and design ideas of the agreement

    Ruan Yifeng, the original author of this paper , The author blog :ruanyifeng.com. 1. introduction HTTP Protocol is one of the most important basic Internet protocols , It evolved from just browsing the web to now , It's the de facto industry standard for short link communications , The latest version HT ...

  6. [ Re posting ] Article to read HTTP/2

    Article to read HTTP/2 http://support.upyun.com/hc/kb/article/1048799/ It's a little shot  •  Published in :2017 year 05 month 18 Japan 15:34:45 •  Updated on :201 ...

  7. [ Re posting ] from HTTP/0.9 To HTTP/2： Article to read HTTP The historical evolution and design ideas of the agreement

    from HTTP/0.9 To HTTP/2: Article to read HTTP The historical evolution and design ideas of the agreement   http://www.52im.net/thread-1709-1-2.html     Ruan Yifeng, the original author of this paper , The author blog :r ...

  8. Article to read clickhouse Cluster monitoring

    More highlights , Please follow the WeChat public account : Back end technology house Article to read clickhouse Cluster monitoring As the saying goes , The army did not arrive , Gateway leading , stay clickhouse Before going to the production environment , We have to work out the relevant monitoring plan , Include metric collection ...

  9. Article to read AI Brief history ： At that time, countries made a wish by burning money , Some of them have not yet been realized

    Article to read AI Brief history : At that time, countries made a wish by burning money , Some of them have not yet been realized Reading guide : In recent days, , Jack ma, . ma . Robin Li and other Internet gangs appeared 2018 World Conference on Artificial Intelligence , And speak on the stage . About the current situation and future of artificial intelligence , They put forward their own views , Also cited ...

  10. stay HUE Import data in text format into hive In the warehouse

    Today, there is a need to put a document in the form of hft And fdd The data of city relationship is imported into hive In the warehouse , Not before hue Do this in ( All of them go through xshell Login fortress machine directly connected to the server for operation ), It is hereby recorded that . - ...

Random recommendation

  1. bignum Big number template

    I accidentally saw a good big number template today , Can be added . reduce . ride . Division and other basic operations , But when subtracting, you can only subtract decimals from large numbers , Negative numbers are not supported , If it's two negative numbers, it's OK to remove the sign and add and then take the opposite , One positive and one negative compare the absolute value , And then subtract . I borrow ...

  2. Git Super practical tutorial

    A piece of git Introductory practical tutorial , Original address http://www.cnblogs.com/tugenhua0707/p/4050072.html One :Git What is it? ? Git It is the most advanced distributed version control system in the world . ...

  3. [ turn ] Mac Next PostgreSQL Installation and use of

    stay mac Next , You can use  homebrew  Direct installation PostgreSQL: 1 brew install postgresql -v Wait a moment ,PostgreSQL  Just install it . Next is the initial database ...

  4. Hadoop learning shuffle The process

    from :http://langyu.iteye.com/blog/992916, Thank you for sharing , Study Hadopp We can pay more attention to performance tuning Shuffle The process is MapReduce At the heart of , Also known as the place where miracles happen , ...

  5. C# Design pattern 14 template method pattern （Template Method）【 Behavior type 】

    One . introduction “ Structural type ” The design pattern of is finished , Let's start today with “ Behavior type ” Design patterns . Now let's start with [ Behavior type ] The first pattern of design patterns , The model is [ Template method ], The English name is :Template Method Patte ...

  6. Create and invoke in project SQLSERVER A simple example of a stored procedure

    Use SQLSERVER Stored procedure can greatly improve the speed of program running , Simplify the difficulty of programming and maintenance , It has been widely used . Create stored procedure It's the same as the data table , You need to create stored procedures before using them , Its concise grammar is : quote : Create PROC ...

  7. Mac adopt brew install reds、memcached

    redis brew install php70-redis The configuration file : /usr/local/etc/php/7.0/conf.d/ext-redis.ini memcached brew inst ...

  8. Nginx The configuration file nginx.conf Detailed explanation of Chinese ( turn ）

    # Definition Nginx Running users and user groups user www www; #nginx Number of processes , It is recommended to set equal to CPU Total core number .worker_processes 8; # Global error log definition type ,[ debug | in ...

  9. JVM Safety point operation and test notes

    JVM Security point learning and code testing Monitoring security points ( Print JVM Pause time , More than GC, Deburring ): -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime Cancel the bias lock : -X ...

  10. [Linux 002]——Linux Common commands

    After the previous study , Have a general understanding of the principles of computer composition and some knowledge of the operating system . Although these knowledge are trivial , Patchwork , In the future work and study, we still need to have a deep understanding . But this preparatory knowledge is important for preparing to step into Linux For the children's shoes of the gate , It should be enough ...