The sparksubmit. Main () method submits external parameters and remotely submits the standalone cluster task

One 、 Official document description

Two 、 introduce pom【 Pay attention to the version problem 】

 <dependency>  
         <groupId>org.apache.spark</groupId>  
         <artifactId>spark-core_2.11</artifactId>  
         <version>2.2.0</version>  
     </dependency>  
  
     <dependency>  
         <groupId>org.apache.hadoop</groupId>  
         <artifactId>hadoop-client</artifactId>  
         <version>2.2.0</version>  
     </dependency>  

3、 ... and 、 Code

import java.io.IOException;

import org.apache.spark.deploy.SparkSubmit;

public class Main {
    public static void main(String[] args) throws IOException {
        String appName = "test java web submit job to spark standalone";
        String className="cfca.xfraud.mltrain.DataTrainning";
        String [] arg0=new String[]{
                "--master","spark://192.168.115.64:7077",//ip port
                 "--deploy-mode","cluster",
                 "--name",appName,
                 "--class",className,// Run main class main
                 "--executor-memory","2G",
                 "--total-executor-cores","10",
                 "--executor-cores","2",
                 "/home/xfraud/ml_train/MachineLearningTraining-0.2.jar",// stay linux Bag on Can be changed to hdfs The path above
                 "LR", "20180817111111", "66"//jar Parameters in , Notice how the parameters are written here
        };
        SparkSubmit.main(arg0);
    }
}

Four 、 Running results

ps.free -m   Check the memory usage on the machine , Pay attention when memory is insufficient ,spark Tasks on the interface will fall into waiting .