当前位置:网站首页>Hudi of data Lake (2): Hudi compilation
Hudi of data Lake (2): Hudi compilation
2022-07-06 00:01:00 【Electro optic flicker】
Catalog
1. Hudi Source package download
2. be based on CentOS7 compile Hudi( Default Spark Etc )
3. be based on windows compile Hudi( Default Spark Etc )
4. be based on windows compile Hudi( choice Spark Etc )
5. be based on CentOS7 compile Hudi( choice Spark Etc )
6. be based on windows compile Hudi Error reporting solutions
7. various Hudi After compiling Jar Package download address
0. Links to related articles
Basic knowledge points of big data A summary of the article
1. Hudi Source package download
because Hudi Unlike Hive、Hadoop And so on , Unzip and install to use ; Instead, you need to download the source code first , Then as needed Hudi Version and integrated components ( Include Spark etc. ) Version to compile ; So you need to download Hudi The source package , Can be in github Download , Download address :GitHub - apache/hudi: Upserts, Deletes And Incremental Processing on Big Data.
2. be based on CentOS7 compile Hudi( Default Spark Etc )
Need to be in centos7 Install in mavne,maven Download the installation tutorial :Maven Download, install and use _yang_shibiao The blog of -CSDN Blog
You can modify CentOS7 For the domestic mirror image , Modify the tutorial :CentOS7 Native yum Modify source to Ali yum Source _yang_shibiao The blog of -CSDN Blog
Upload the corresponding zip package , Then decompress , Decompress it as follows :
notes : What the blogger downloaded is 0.10.1 edition , The root directory has been renamed , And because bloggers have compiled , So there are packaging package
Carry out orders , Compile , Compile command : mvn package -DskipTests
Because relevant files will be downloaded during compilation maven Dependency package , So according to the above maven Download and install as shown in the blog , Configure domestic image , Increase download speed , The specific compilation time depends on the machine and network , A total of 1.5G Left and right jar package , Bloggers have been compiled many times , stay maven There is nothing in the warehouse Jar A total of 6 Hours , When all Jar There is recompilation ,7 Minutes to compile , After compilation, see the figure below :
3. be based on windows compile Hudi( Default Spark Etc )
Need to be in windows Install in maven, And configure the corresponding environment variables , You can refer to maven Download the installation tutorial :Maven Download, install and use _yang_shibiao The blog of -CSDN Blog
from github Download the corresponding hudi package , Then unzip the package , The content shown in the following figure will be generated :
Execute under the above directory maven The compile command of : mvn package -DskipTests
front 30 Multiple packages can be compiled normally , Will find hudi-integ-test Package compilation failed , As shown in the figure below :
This is because it is in windows Environment ,hudi-integ-test The module will execute linux In the environment bash Script , Thus the execution fails , This module needs to be commented out , Perform discovery again packaging/hudi-integ-test-bundle Module will reference hudi-integ-test The content of this module , So two test Modules are commented out , As shown in the figure below :
Execute the compilation script again , The successful running , As shown in the figure below : mvn package -DskipTests
4. be based on windows compile Hudi( choice Spark Etc )
Directly execute the following compilation commands , You can select the corresponding Spark Version and Scala Version, etc : mvn clean package -DskipTests -Dspark3 -Dscala-2.12
5. be based on CentOS7 compile Hudi( choice Spark Etc )
Directly execute the following compilation commands , You can select the corresponding Spark Version and Scala Version, etc : mvn clean package -DskipTests -Dspark3 -Dscala-2.12
6. be based on windows compile Hudi Error reporting solutions
As mentioned above 3 section be based on windows compile Hudi( Default Spark Etc ) Shown , Because hudi-integ-test The module will execute linux In the environment bash Script , Need to put hudi-integ-test Module annotation , Perform discovery again packaging/hudi-integ-test-bundle Module will reference hudi-integ-test The content of this module , So two test Modules are commented out , So you can do it windows To compile .
7. various Hudi After compiling Jar Package download address
To be updated ...
notes :Hudi The series of blog posts are through Hudi Written in the official website learning records , One of them is to add personal understanding , If there is any deficiency , Please understand
notes : Links to other related articles go here ( Include Hudi Blog posts related to big data, including ) -> Basic knowledge points of big data A summary of the article
边栏推荐
- QT--线程
- C reflection and type
- FFT learning notes (I think it is detailed)
- Upgrade openssl-1.1.1p for openssl-1.0.2k
- How to rotate the synchronized / refreshed icon (EL icon refresh)
- 7.5 decorator
- MySQL之函数
- 转:未来,这样的组织才能扛住风险
- [day39 literature extensive reading] a Bayesian perspective on magnetic estimation
- 【NOI模拟赛】Anaid 的树(莫比乌斯反演,指数型生成函数,埃氏筛,虚树)
猜你喜欢
单商户V4.4,初心未变,实力依旧!
Senparc. Weixin. Sample. MP source code analysis
Gd32f4xx UIP protocol stack migration record
Breadth first search open turntable lock
Fiddler Everywhere 3.2.1 Crack
用列錶初始化你的vector&&initializer_list簡介
亲测可用fiddler手机抓包配置代理后没有网络
提升工作效率工具:SQL批量生成工具思想
Upgrade openssl-1.1.1p for openssl-1.0.2k
Learn PWN from CTF wiki - ret2libc1
随机推荐
教你在HbuilderX上使用模拟器运行uni-app,良心教学!!!
FFmpeg学习——核心模块
openssl-1.0.2k版本升级openssl-1.1.1p
【NOI模拟赛】Anaid 的树(莫比乌斯反演,指数型生成函数,埃氏筛,虚树)
[gym 102832h] [template] combination lock (bipartite game)
USB Interface USB protocol
QT -- thread
用列錶初始化你的vector&&initializer_list簡介
【SQL】各主流数据库sql拓展语言(T-SQL 、 PL/SQL、PL/PGSQL)
多普勒效應(多普勒頻移)
Fiddler Everywhere 3.2.1 Crack
What if the C disk is not enough? Let's see how I can clean up 25g of temp disk space after I haven't redone the system for 4 years?
上门预约服务类的App功能详解
Yunna | what are the main operating processes of the fixed assets management system
Initialize your vector & initializer with a list_ List introduction
Bao Yan notebook IV software engineering and calculation volume II (Chapter 8-12)
【GYM 102832H】【模板】Combination Lock(二分图博弈)
云呐|固定资产管理系统功能包括哪些?
After summarizing more than 800 kubectl aliases, I'm no longer afraid that I can't remember commands!
Choose to pay tribute to the spirit behind continuous struggle -- Dialogue will values [Issue 4]