Data Lake (20): Flink is compatible with iceberg, which is currently insufficient, and iceberg is compared with Hudi

Flink compatible Iceberg Current deficiencies and Iceberg And Hudi contrast

One 、Flink compatible Iceberg Not enough at present

• Iceberg Currently not supported Flink SQL Query metadata information of table , Need to use Java API Realization .
• Flink Creating with hidden partitions is not supported Iceberg surface
• Flink Do not support with WaterMark Of Iceberg surface
• Flink Adding columns is not supported 、 Delete column 、 Rename column operation .
• Flink Yes Iceberg Connector Support is not perfect .

Two 、Iceberg And Hudi contrast

Iceberg and Hudi It's all data Lake Technology , From the perspective of community activity ,Iceberg There is transcendence Hudi The trend of . They have the following in common ：

• Both are data organization methods based on storage formats
• Provide ACID Ability , Provide certain transactions 、 Parallel execution capability
• Provide row level data modification capability .
• Provide a certain amount of Schema Expand capabilities , for example ： newly added 、 modify 、 Delete column operation .
• Support data consolidation , Working with small files .
• Support Time travel Query snapshot data .
• Support batch and real-time data reading and writing

Iceberg And Hudi The difference between them lies in the following points ：

• Iceberg Support Parquet、avro、orc data format ,Hudi Support Parquet and Avro Format .
• The data storage and query mechanisms of the two are different

Iceberg Only one table storage mode is supported , There is metadata file、manifest file and data file Form a storage structure , When querying, first find Metadata The metadata is then filtered to find the corresponding SnapShot Corresponding manifest files , Then find the corresponding data file .Hudi Two table storage modes are supported ：Copy On Write（ Merge on write ） and Merge On Read（ Merge while reading ）, When querying, directly read the corresponding snapshot data .

• When dealing with small file merging ,Iceberg Only support API Method to manually process and merge small files ,Hudi For small files, merge processing can be performed automatically according to the configuration .
• Spark And Iceberg and Hudi Integration time ,Iceberg Yes SparkSQL At present, our support is better .Spark And Hudi Integration is more Spark DataFrame API operation .
• About Schema aspect ,Iceberg Schema It is decoupled from the computing engine , Do not rely on any computing engine , and Hudi Of Schema Rely on the computing engine Schema.