当前位置：网站首页>Spark Learning: compile spark source code in win10

Spark Learning: compile spark source code in win10

2022-07-24 09:37:00 【I love evening primrose a】

compile Spark Source code

One 、 Environmental preparation

1、JDK1.8
2、scala 2.12.15
3、maven 3.8.5
3、Spark3.2.1
spark Download the latest sourcecode
Insert picture description here

Two 、 compile

1、 Open source parent pom.xml modify scala/maven/java Corresponding version
Insert picture description here

2、 To configure maven Mirror source
Alibaba cloud
3、 stay cmd/powershell Move to the source directory parent pom.xml In the directory

mvn -DskipTests clean package
// Use mvn Skip unit test compilation

Insert picture description here

3、 ... and 、IDE Import execution SparkPi.scala

1、 stay windows Debugging in environment Spark Program , Need to use winutils.exe, Here is a reference to the following article
Windows debugging 、 Development Spark Program
2、 Report errors NoClassDefFoundError:scala/collection/Seq
The reason is just generated spark The related dependent package was not found , hold assembly\target\scala-2.11\jar Lower generation spark The required jar Package is added to the examples Module in
Insert picture description here
3、 Report errors Could not find spark-version-info.properties
newly build spark-version-info.properties To examples Of resources Catalog , Add version information to the file , As shown below

in addition , You need to specify the deployment mode and main class
Can be local mode :-Dspark.master=local
It can also be a machine on a virtual machine :-Dspark.master=spark://192.168.127.108:7077
Insert picture description here
perfect ！