[PostgreSQL] - Storage structure and cache shared_buffers

一、数据存储

1、物理存储位置

-- The location on disk where the lookup table is located
select pg_relation_filepath('表名')
-- out
base/16393/24804

复制

包含三个文件, 具体介绍请看PostgreSQL目录结构之base目录,Free Space Map and Visibility Map:

24804代表表的OID(object id)

• 24804 - 数据和索引文件(超过1G每1GDivide a section, 命名*.1, *.2)
• 24804_fsm(free space map): 空闲空间映射,Primarily used to keep track of the space available in the relationship.
• 24804_vm(visibility map): 可见性映射, Each heap relationship has a visibility map（VM）Used to track which pages Contains only tuples known to be visible to all active transactions,It also tracks which pages only contain An unfrozen tuple.

注意: Because a page is8K,If a table's columns may store quite large items,Then the table will have one associated with itTOAST表, It is used for off-line storage of field values ​​that cannot be kept in the table row. 参考68.2. TOAST.

2、物理存储结构

每个表每8K分一页,Row data exists on the page

-- Lookup table pages、行数
SELECT relpages as 页个数, reltuples as 行个数 FROM pg_class WHERE relname = '表名'

-- The size of the query space(排名前20)
SELECT table_schema || '.' || table_name                                                         AS table_full_name,
       pg_size_pretty(pg_total_relation_size('"' || table_schema || '"."' || table_name || '"')) AS size
FROM information_schema.tables
ORDER BY pg_total_relation_size('"' || table_schema || '"."' || table_name || '"') DESC
limit 20;

复制

A page in the table contains three types of data described below：

1. heap tuple(s) ——The heap tuple itself is a record of data.They are stacked in order starting from the bottom of the page.
2. line pointer(s) – A row pointer has 4 个字节长,and holds a pointer to each heap tuple.它也称为Project pointer. Row pointers form a simple array,Plays the role of a tuple index.each index from 1 开始按顺序编号,称为offset number.When a new tuple is added to the page,A new row pointer is also pushed onto the array to point to the new one.
3. header data——由结构PageHeaderDataThe defined header data is allocated at the beginning of the page.它长 24 字节,Contains general information about the page.The main variables of the structure are described below.

• pd_lsn——This variable stores the last change written to this page XLOG 记录的 LSN.它是一个 8 字节的无符号整数,与 WAL（Write-Ahead Logging）机制有关.
• pd_checksum – This variable stores the checksum value for this page.
• pd_lower, pd_upper – pd_lower Pointer to end of line,pd_upper Points to the beginning of the latest heap tuple.
• pd_special – 此变量用于索引.in the page in the table,It points to the end of the page.（in pages within the index,它指向特殊空间的开头,That is, only the data area held by the index,According to the kind of index type,如B-tree、GiST、GiNetc. contain specific data.）

二、检索

1、数据扫描方式

左侧为顺序扫描,右侧为b-tree索引扫描

b-tree扫描细节

三、缓存cache

参考: 深入理解Postgres中的cache

1、概述

我们知道,大多数OLTPWorkloads are randomI/O,But getting from disk is very slow.为了克服这个问题,Similar to other existing database systems,PostgresAlso cache the data toRAM(That is what we call memory)以提高性能.

2、缓存 shared_buffers

shared_buffers所代表的内存区域可以看成是一个以8KB的block为单位的数组,即最小的分配单位是8KB.这正好是一个page的大小,每个page以page内部的元数据(Page Header)distinguish each other. 这样,当Postgres想要从disk获取(主要是table和index)数据(page)时,他会(根据page的元数据)先搜索shared_buffers,确认该page是否在shared_buffers中,如果存在,则直接命中,返回缓存的数据以避免I/O. 如果不存在,Postgres才会通过I/O访问disk获取数据(Obviously than fromshared_buffersfetching is much slower).

3、缓存淘汰

以页为单位,cache满的时候,Infrequently used pages are eliminated.The eliminated data will be refreshed,But the general data is passedWAL+CheckpointerEnsure that the modified data is flushed,而不用等到cacheWhen it is full, flush the disk.

4、How to observe whether to gocache

使用explain时,Shared read表示来自disk,Shared hitis already therecache中

Graphical analysis is also shown

image.png

5、查看缓存情况

SELECT
     c.relname,
     pg_size_pretty(count(*) * 8192) as buffered,
     round(100.0 * count(*) / (SELECT setting FROM pg_settings WHERE name='shared_buffers')::integer, 1) AS buffer_percent,
     round(100.0 * count(*) * 8192 / pg_table_size(c.oid), 1) AS percent_of_relation
 FROM pg_class c
 INNER JOIN pg_buffercache b ON b.relfilenode = c.relfilenode
 INNER JOIN pg_database d ON (b.reldatabase = d.oid AND d.datname = current_database())
 GROUP BY c.oid, c.relname
 ORDER BY 3 DESC
 LIMIT 10;

复制

6、Cache is invalidated

参考: PostgreSQL查询shared bufferUsage and cleaning methods

7、Cache flushing strategy

写操作先写WAL日志,After the write is successful, write the memory data,之后CheckpointerPerform data flushing,Cached data is the latest data.

四、What lessons can be learned for us to optimize queries

1、The key table data is resident in the cache

参考: PostgreSQL-Cache tool 将重点数据、Data with high real-time requirementsManually loaded into the database cache,Reduce memory elimination for key data,不稳定IO.At the same time, the issue of brushing must be considered,Downtime disaster recovery problem. This also sacrifices performance for the rest of the less critical functions,Need to actually de-stress.

2、The computing system and the query system are separated from master and slave

计算服务-写主 Report reading service-读从

• 优势: 1、Master-slave service because the behavior is different,So the hotspot data is different,Isolation ensures that their respective cache hit rates are higher.
• 其他影响: 1、Even if there is a delay in the master and slave,It is also that the running data is not generated in time,As long as the delay is not too high,The impact is not great. 2、系统变复杂,Reduced ease of maintenance.

3、单测、During stress testing, database cache interference should be excluded

排除干扰,Or bring cache synthesis to consider optimization solutions.

五、other optimization directions

1、PREPARE预加载,PostgreSQL手册-prepare 2、PostgreSQL 列存索引 - 新方式 - 列存 3、介绍PostgreSQL CTE(common table expressions) - with as 简化

参考

1、深入理解Postgres中的cache 2、PostgreSQLIntroduction to physical storage 3、PostgreSQL Internals Through Pictures 4、The Internals of PostgreSQL