Data Lake (2): What is Hudi

​What is Hudi

Apache Hudi is an open source solution for Data Lakes. Hudi is short for Hadoop Updates and Incrementals. It is a Data Lakes solution developed and open sourced by Uber.Hudi can manage large-scale analytical data sets based on HDFS, and can perform operations such as inserting, updating, and incremental consumption of data. The main purpose is to efficiently reduce the data delay in the ingestion process.

Hudi is very lightweight and can be integrated with Spark and Flink as a lib

Hudi official website:

https://hudi.apache.org

Hudi is based on Parquet column storage and Avro row storage, while avoiding the creation of small files to achieve high-efficiency and low-latency data access.Provides insert updates, incremental pulls, and full pulls on HDFS datasets.Hudi has the following features:

• Quick upsert, can insert index.

• Atomically manipulate data with rollback.

• Snapshot isolation between writers and queries.

• The savepoint for data recovery.Hudi implements data recovery through Savepoint.

• Manage file size, use statistics layout.

• Asynchronous compression of row and column data.

​