Hudi of data Lake (2): Hudi compilation

Catalog

0. Links to related articles

1. Hudi Source package download

2. be based on CentOS7 compile Hudi（ Default Spark Etc ）

3. be based on windows compile Hudi（ Default Spark Etc ）

4. be based on windows compile Hudi（ choice Spark Etc ）

5. be based on CentOS7 compile Hudi（ choice Spark Etc ）

6. be based on windows compile Hudi Error reporting solutions

7. various Hudi After compiling Jar Package download address

---

0. Links to related articles

Basic knowledge points of big data A summary of the article

1. Hudi Source package download

         because Hudi Unlike Hive、Hadoop And so on , Unzip and install to use ; Instead, you need to download the source code first , Then as needed Hudi Version and integrated components （ Include Spark etc. ） Version to compile ; So you need to download Hudi The source package , Can be in github Download , Download address ：GitHub - apache/hudi: Upserts, Deletes And Incremental Processing on Big Data.

2. be based on CentOS7 compile Hudi（ Default Spark Etc ）

Need to be in centos7 Install in mavne,maven Download the installation tutorial ：Maven Download, install and use _yang_shibiao The blog of -CSDN Blog

You can modify CentOS7 For the domestic mirror image , Modify the tutorial ：CentOS7 Native yum Modify source to Ali yum Source _yang_shibiao The blog of -CSDN Blog

Upload the corresponding zip package , Then decompress , Decompress it as follows ：

notes ： What the blogger downloaded is 0.10.1 edition , The root directory has been renamed , And because bloggers have compiled , So there are packaging package

Carry out orders , Compile , Compile command ： mvn package -DskipTests

Because relevant files will be downloaded during compilation maven Dependency package , So according to the above maven Download and install as shown in the blog , Configure domestic image , Increase download speed , The specific compilation time depends on the machine and network , A total of 1.5G Left and right jar package , Bloggers have been compiled many times , stay maven There is nothing in the warehouse Jar A total of 6 Hours , When all Jar There is recompilation ,7 Minutes to compile , After compilation, see the figure below ：

3. be based on windows compile Hudi（ Default Spark Etc ）

Need to be in windows Install in maven, And configure the corresponding environment variables , You can refer to maven Download the installation tutorial ：Maven Download, install and use _yang_shibiao The blog of -CSDN Blog

from github Download the corresponding hudi package , Then unzip the package , The content shown in the following figure will be generated ：

Execute under the above directory maven The compile command of ： mvn package -DskipTests

front 30 Multiple packages can be compiled normally , Will find hudi-integ-test Package compilation failed , As shown in the figure below ：

This is because it is in windows Environment ,hudi-integ-test The module will execute linux In the environment bash  Script , Thus the execution fails , This module needs to be commented out , Perform discovery again packaging/hudi-integ-test-bundle Module will reference hudi-integ-test The content of this module , So two test Modules are commented out , As shown in the figure below ：

Execute the compilation script again , The successful running , As shown in the figure below ： mvn package -DskipTests

4. be based on windows compile Hudi（ choice Spark Etc ）

Directly execute the following compilation commands , You can select the corresponding Spark Version and Scala Version, etc ： mvn clean package -DskipTests -Dspark3 -Dscala-2.12  

5. be based on CentOS7 compile Hudi（ choice Spark Etc ）

Directly execute the following compilation commands , You can select the corresponding Spark Version and Scala Version, etc ： mvn clean package -DskipTests -Dspark3 -Dscala-2.12  

6. be based on windows compile Hudi Error reporting solutions

As mentioned above 3 section   be based on windows compile Hudi（ Default Spark Etc ） Shown , Because hudi-integ-test The module will execute linux In the environment bash Script , Need to put hudi-integ-test Module annotation , Perform discovery again packaging/hudi-integ-test-bundle Module will reference hudi-integ-test The content of this module , So two test Modules are commented out , So you can do it windows To compile .

7. various Hudi After compiling Jar Package download address

To be updated ...

---

notes ：Hudi The series of blog posts are through Hudi Written in the official website learning records , One of them is to add personal understanding , If there is any deficiency , Please understand

notes ： Links to other related articles go here （ Include Hudi Blog posts related to big data, including ） ->  Basic knowledge points of big data A summary of the article

---