Hudi compilation of data Lake architecture

Preface

Speaking of compilation hudi, After the first pass , Let's go back , Discovery is the first time you are not familiar with , All the problems can be summarized as maven The warehouse is not configured properly . At the beginning, I just configured the Alibaba cloud warehouse , However, errors are constantly reported later , Then Baidu and Google find the reason , Then adjust the configuration , To compile , In the end, it worked , So the overall compilation is not complicated , As long as the configuration is correct , Then I will post the configuration that can be passed at last , This is where I think I can help most of my classmates .

Version and source code

hudi The iteration is still relatively fast , Because it also depends on hadoop and spark, In order to combine , I'm using 0.9.0 edition , Corresponding address ：[https://hudi.apache.org/releases/release-0.9.0](https://hudi.apache.org/releases/release-
0.9.0)
The source code can be clicked download Part of , You can download it.

Environmental preparation

[[email protected] target]# mvn -v
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /apps/svr/maven
Java version: 1.8.0_144, vendor: Oracle Corporation, runtime: /apps/svr/jdk1.8.0_144/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.5-3.el6.x86_64", arch: "amd64", family: "unix"

compile

The compilation process is just normal maven project , It's not very complicated

mvn clean install -DskipTests -DskipITs -Dscala-2.12 -Dspark3 

As mentioned earlier , Success comes after success , The final result , I still need to make an appearance :

[INFO] Hudi 0.9.0 ......................................... SUCCESS [  2.758 s]
[INFO] hudi-common ........................................ SUCCESS [ 20.652 s]
[INFO] hudi-timeline-service .............................. SUCCESS [  3.375 s]
[INFO] hudi-client ........................................ SUCCESS [  0.208 s]
[INFO] hudi-client-common ................................. SUCCESS [ 12.704 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [  5.637 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 24.567 s]
[INFO] hudi-sync-common ................................... SUCCESS [  1.197 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [  6.125 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.107 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 13.649 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 11.451 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 45.515 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 23.751 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 51.554 s]
[INFO] hudi-cli ........................................... SUCCESS [ 32.192 s]
[INFO] hudi-java-client ................................... SUCCESS [  3.458 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 12.356 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 17.489 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [  3.055 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.131 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  6.229 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [  2.009 s]
[INFO] hudi-spark3-bundle_2.12 ............................ SUCCESS [ 14.460 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [  9.588 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [  7.371 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [  0.852 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [01:08 min]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  0.178 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  0.124 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [  0.112 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [  0.541 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [  0.121 s]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [  0.128 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  0.181 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [  0.171 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [  0.231 s]
[INFO] hudi-integ-test .................................... SUCCESS [01:05 min]
[INFO] hudi-integ-test-bundle ............................. SUCCESS [02:29 min]
[INFO] hudi-examples ...................................... SUCCESS [  9.459 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [  9.703 s]
[INFO] hudi-flink-bundle_2.12 0.9.0 ....................... SUCCESS [ 24.891 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:02 min
[INFO] Finished at: 2022-05-04T00:53:46+08:00
[INFO] ------------------------------------------------------------------------

The key maven To configure

This configuration is constantly reporting errors , Find the information and finally solve it , So when you compile, you can directly put my configuration in settings.xml Just inside .

 <mirrors>
 <mirror>
    <id>alimaven</id>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>aliyunmaven</id>
    <mirrorOf>*</mirrorOf>
    <name> Alibaba cloud spring plug-in unit </name>
    <url>https://maven.aliyun.com/repository/spring-plugin</url>
</mirror>
<mirror>
    <id>repo2</id>
    <name>Mirror from Maven Repo2</name>
    <url>https://repo.spring.io/plugins-release/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>UK</id>
    <name>UK Central</name>
    <url>http://uk.maven.org/maven2</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>jboss- public-repository- group</id>
    <name>JBoss Public Repository Group</name>
    <url>http://repository.jboss.org/nexus/content/groups/public</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>CN</id>
    <name>OSChina Central</name>
    <url>http: //maven.oschina.net/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>google-maven-centralk</id>
    <name>GCS Maven Central mirror Asia Pacific</name>
    <url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>confluent</id>
    <name>confluent maven</name>
    <url>http://packages.confluent.io/maven/</url>
    <mirrorOf>confluent</mirrorOf>
</mirror>
<mirror>
    <id>jboss- public-repository-group</id>
    <name>JBoss Public Repository Group</name>
    <url>http://repository.jboss.org/nexus/content/groups/public</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>CN</id>
    <name>OSChina Central</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>google-maven- centralk</id>
    <name>GCS Maven Central mirror Asia Pacific</name>
    <url>https://maven-central-asia.storage-download.googleapis. com/maven2/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>confluent</id>
    <name>confluent maven</name>
    <url>http://packages.confluent.io/maven/</url>
    <mirrorOf>confluent</mirrorOf>
</mirror>
 </mirrors>

Postscript

hudi Compilation will be much simpler than others ~~