Spark3.3.0源码编译补充篇-抓狂的证书问题

前言

现在是2022年6月18日9:50，凌晨倒腾快到2点了，今天其实是京东的店庆，另外我还记得今天其实是中考的日子，不知道是不是中学都是这天毕业，再或者是学校不一样，时间点不一样，可能时间也有调整了吧。事情是这样，我本身是为了搭建hudi的环境，所以要要搭建Spark环境，Hadoop环境，奈何家里网速有点慢，我找了官网，Spark3.3.0源码其实是28M，但是那个安装包要261M，其实Hadoop2.7.3源码是17.3M但是按照包200多M，最近也是网速抽风，我下载不过来，所以准备一边下载一边自己编译的搞法，Hadoop比较顺利，但是我发现Spark3.3版本和3.0版本按照我之前的搞法死活过不去。我原来写过一篇文章 Spark3.0源码编译打包，但是到了3.3.0需要再修正了，所以就决定再次记录下来。

先晒结果

国内网络不通确实是比较伤脑筋的，以下的结果是倒腾了蛮久才出现的，起因是在Spark3.3.0中增加了谷歌仓库插件的访问，纠结了很久才调试通过。

Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-project/1.8/leveldbjni-project-1.8.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-project/1.8/leveldbjni-project-1.8.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/fusesource-pom/1.9/fusesource-pom-1.9.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/fusesource-pom/1.9/fusesource-pom-1.9.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.pom (5.5 kB at 7.1 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-base/2.13.3/jackson-base-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-base/2.13.3/jackson-base-2.13.3.pom (9.9 kB at 14 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-bom/2.13.3/jackson-bom-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-bom/2.13.3/jackson-bom-2.13.3.pom (17 kB at 24 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-parent/2.13/jackson-parent-2.13.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-parent/2.13/jackson-parent-2.13.pom (7.4 kB at 9.3 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/oss-parent/43/oss-parent-43.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/oss-parent/43/oss-parent-43.pom (24 kB at 30 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.pom (16 kB at 21 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.pom (6.1 kB at 7.6 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (14 kB at 19 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j/2.17.2/log4j-2.17.2.pom

抓狂的证书问题

一直连接不上的仓库

编译命令还是没有变化

./dev/make-distribution.sh --name spark-3.3.0  --tgz  -Phadoop-2 -Dhadoop.version=2.7.4  -Phive -Phive-thriftserver  -Pyarn 

问题是出现在pom.xml中，有这么部分

<pluginRepository>
      <id>gcs-maven-central-mirror</id>
      <!--
        Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
        See https://storage-download.googleapis.com/maven-central/index.html
      -->
      <name>GCS Maven Central mirror</name>
      <!--<url>https://maven-central.storage-download.googleapis.com/maven2/</url>-->
	  <url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </pluginRepository>

网络上是叫我改成阿里的地址，我试了，其实没有用，依赖库在ali的仓库里面是没有的，然后根据提示，我换成了谷歌的亚洲仓库，也不起作用，日志一直卡在下载maven-metadata.xml的地方，完全超时

gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/maven2/): transfer failed for https://maven-central.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

一直报错

https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

比较长时间不断在切换仓库，以为就是googleapis连接不上导致的.

转折点

也不知道哪里来的想法，我突然想确定一下是不是确实连接访问不了,我直接用wget去访问了一下

wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

结果报错了，不过内容并不是连接不上，而是证书问题

[[email protected] spark-3.3.0]# wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml
--2022-05-04 12:31:00--  https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml
正在解析主机 maven-central-asia.storage-download.googleapis.com... 142.251.43.16
正在连接 maven-central-asia.storage-download.googleapis.com|142.251.43.16|:443... 已连接。
错误: 无法验证 maven-central-asia.storage-download.googleapis.com 的由 “/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3” 颁发的证书:
  颁发的证书还未生效。
错误: 证书通用名 “*.storage.googleapis.com” 与所要求的主机名 “maven-central-asia.storage-download.googleapis.com” 不符。
要以不安全的方式连接至 maven-central-asia.storage-download.googleapis.com，使用‘--no-check-certificate’。
You have new mail in /var/spool/mail/root

这个问题好办，有经验,修改一些不校验的参数

wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml --no-check-certifica

maven命令行修改参数

找到了原因，就想办法在maven中去掉限制,这个也是之前知道，所以就直接加上
-Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true，这样可以不进行证书校验

./dev/make-distribution.sh --name spark-3.3.0  --tgz -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true -Phadoop-2 -Dhadoop.version=2.7.4  -Phive -Phive-thriftserver  -Pyarn 

网络问题就可以顺利通过了

内存问题

最后还是需要调整一下内存

export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"