Spark3.3.0 source code compilation supplement - Crazy certificate problem

Preface

Now it is 2022 year 6 month 18 Japan 9:50, It's almost midnight 2 O 'clock , Today is actually Jingdong's store celebration , In addition, I still remember that today is actually the day of the high school entrance examination , I don't know if all middle schools graduate on this day , Or the school is different , The time point is different , Maybe the time has been adjusted . It's like this , I am trying to build hudi Environment , So we should build Spark Environmental Science ,Hadoop Environmental Science , However, the Internet speed at home is a little slow , I found the official website ,Spark3.3.0 The source code is actually 28M, But that installation package needs 261M, Actually Hadoop2.7.3 The source code is 17.3M But according to the package 200 many M, Recently, the Internet speed is also very high , I can't download it , So I'm going to download it and compile it myself ,Hadoop Quite smooth , But what I found was that Spark3.3 Version and 3.0 The version can't survive as I did before . I once wrote an article Spark3.0 Source compilation package , But here we are 3.3.0 It needs to be revised , So I decided to write it down again .

Sun the results first

It is really troublesome that the domestic network is blocked , The following results are the result of tossing for a long time , The reason is that Spark3.3.0 Added access to Google warehouse plug-ins , It took a long time to debug .

Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-project/1.8/leveldbjni-project-1.8.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-project/1.8/leveldbjni-project-1.8.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/fusesource-pom/1.9/fusesource-pom-1.9.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/fusesource/fusesource-pom/1.9/fusesource-pom-1.9.pom (0 B at 0 B/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.pom (5.5 kB at 7.1 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-base/2.13.3/jackson-base-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-base/2.13.3/jackson-base-2.13.3.pom (9.9 kB at 14 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-bom/2.13.3/jackson-bom-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-bom/2.13.3/jackson-bom-2.13.3.pom (17 kB at 24 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-parent/2.13/jackson-parent-2.13.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/jackson-parent/2.13/jackson-parent-2.13.pom (7.4 kB at 9.3 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/oss-parent/43/oss-parent-43.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/oss-parent/43/oss-parent-43.pom (24 kB at 30 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.pom (16 kB at 21 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.pom (6.1 kB at 7.6 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom
Downloaded from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-api/2.17.2/log4j-api-2.17.2.pom (14 kB at 19 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j/2.17.2/log4j-2.17.2.pom

Crazy certificate problem

A warehouse that has not been connected

The compile command is still unchanged

./dev/make-distribution.sh --name spark-3.3.0  --tgz  -Phadoop-2 -Dhadoop.version=2.7.4  -Phive -Phive-thriftserver  -Pyarn 

The problem is that pom.xml in , There is such a part

<pluginRepository>
      <id>gcs-maven-central-mirror</id>
      <!--
        Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
        See https://storage-download.googleapis.com/maven-central/index.html
      -->
      <name>GCS Maven Central mirror</name>
      <!--<url>https://maven-central.storage-download.googleapis.com/maven2/</url>-->
	  <url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </pluginRepository>

On the Internet, I was asked to change my address to Ali , I tried , It doesn't work , The dependency library is ali There is nothing in the warehouse , Then follow the prompts , I switched to Google's Asian warehouse , It doesn't work , Logs have been stuck downloading maven-metadata.xml The place of , Full timeout

gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/maven2/): transfer failed for https://maven-central.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

Has been an error

https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

They have been switching warehouses for a long time , I thought it was googleapis Not connected .

The turning point

I don't know where the idea came from , I suddenly want to make sure that I can't access the connection , I use... Directly wget Went to visit

wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml

It turned out to be wrong , But the content is not disconnected , It's about certificates

[[email protected] spark-3.3.0]# wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml
--2022-05-04 12:31:00--  https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml
 Resolving host  maven-central-asia.storage-download.googleapis.com... 142.251.43.16
 on connection  maven-central-asia.storage-download.googleapis.com|142.251.43.16|:443...  Connected .
 error :  Can't verify  maven-central-asia.storage-download.googleapis.com  By  “/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3”  Certificate issued :
   The certificate issued is not yet valid .
 error :  Certificate common name  “*.storage.googleapis.com”  With the required host name  “maven-central-asia.storage-download.googleapis.com”  Not in conformity with .
 To connect to... In an unsafe manner  maven-central-asia.storage-download.googleapis.com, Use ‘--no-check-certificate’.
You have new mail in /var/spool/mail/root

This problem is easy to handle , Experienced , Modify some parameters that are not verified

wget https://maven-central-asia.storage-download.googleapis.com/maven2/org/apache/maven/plugins/maven-metadata.xml --no-check-certifica

maven Command line modification parameters

Found the reason , Just find a way to maven Remove restrictions from , This is also known before , So just add
-Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true, In this way, certificate verification is not required

./dev/make-distribution.sh --name spark-3.3.0  --tgz -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true -Phadoop-2 -Dhadoop.version=2.7.4  -Phive -Phive-thriftserver  -Pyarn 

The network problem can be solved smoothly

Memory problems

Finally, you need to adjust the memory

export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"