Data Lake (VIII): Iceberg data storage format

 Iceberg Data storage format

One 、Iceberg The term

• data files（ Data files ）:

The data file is Apache Iceberg Tables are files that actually store data , It's usually in the data storage directory of the table data Under the table of contents , If our file format is parquet, So the document is based on “.parquet” ending , for example ：

00000-0-root_20211212192602_8036d31b-9598-4e30-8e67-ce6c39f034da-job_1639237002345_0025-00001.parquet  It's just a data file .

Iceberg Each update produces multiple data files (data files).

• Snapshot（ Table snapshot ）：

A snapshot represents the state of a table at a certain time . Each snapshot will list all the data in the table at a certain time data files list .data files It's stored in different manifest files Inside ,manifest files Is stored in a Manifest list In the document , And one Manifest list The file represents a snapshot .

• Manifest list（ List of checklists ）：

manifest list Is a metadata file , It lists the snapshot of the build table （Snapshot） List of （Manifest file）. What is stored in this metadata file is Manifest file list , Every Manifest file Occupy a line . Each row stores Manifest file The path of 、 Its stored data file （data files） Partition range , Added several number files 、 Deleted several data files and other information , This information can be used to provide filtering when querying , Speed up .

• Manifest file（ Inventory file ）：

Manifest file It is also a metadata file , It lists the components of the snapshot （snapshot） Data files for （data files） List information for . Each line is a detailed description of each data file , Including the status of the data file 、 File path 、 Zone information 、 Statistics at the column level （ For example, the maximum and minimum of each column 、 Null number, etc ）、 The size of the file and the number of data lines in the file . Column level statistics can filter out unnecessary files when scanning table data .

Manifest file In order to avro Format for storage , With “.avro” The suffix ends , for example ：8138fce4-40f7-41d7-82a5-922274d2abba-m0.avro.

Two 、 Table format Table Format

Apache Iceberg As a data Lake solution , It is an open table format for large analysis data sets (Table Format), Table format can be understood as an organization of metadata and data files .Iceberg The underlying data store can be docked HDFS,S3 file system , And supports a variety of file formats , In the calculation frame （Spark、Flink） under , Data files .

​

Here's how Iceberg How the underlying files are organized , The picture below is Iceberg Middle table format ,s0、s1 It represents the table Snapshot Information , Each represents a snapshot of the current operation , Every time commit Will generate a snapshot Snapshot, Every Snapshot The snapshot corresponds to a manifest list Metadata file , Every manifest list Contains multiple Manifest Metadata file ,manifest The file address corresponding to the data generated by the current operation is recorded in , That is to say data file The address of .

be based on snapshot Management style ,Iceberg Can get the historical version data of the table 、 Incremental read operation on table ,data files Storage supports different file formats , At present, we support parquet、ORC、Avro Format .

​

About Iceberg Table data underlying organization details , You can pay attention to the following articles , I will explain it in detail .