Idea remote debugging jar submitted by spark submit

1、 Edit code

object WordCount {

  def main(args: Array[String]): Unit = {

    val sc = new SparkContext(new SparkConf().setAppName("WordCount "))

    val rdd: RDD[String] = sc.makeRDD(List(
      "spark hello", "hive", "hadoop hbase", "spark hadoop", "hbase"
    ))

    val value: RDD[String] = rdd.flatMap(_.split(" "))

    val mapRDD: RDD[(String, Int)] = value.map((_, 1))

    val result: RDD[(String, Int)] = mapRDD.reduceByKey(_ + _)

    result.collect().foreach(println)

  }

}

2、 open IDEA To configure , Configure remote connection

  There are two main configurations , One is host, One is port. But I circled three places in the picture , The last place is to read the English description carefully , That string of commands is We need to copy and add to spark-submit In the parameters of the submit command Of .

 3、 modify spark-submit, Add remote connection parameters

spark-submit \
--master spark://bigdata.node2:7077 \
--class test.wordcount.WordCount \
--num-executors 1 \
--driver-memory 499m \
--driver-cores 1 \
--executor-memory 499m \
--executor-cores 1 \
--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8888 \
/opt/spark-jar/wyt01bigdata-1.0-SNAPSHOT.jar

You can see , We will IDEA Remote commands generated in -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8888 Added in spark Submit command . Here is a parameter to pay special attention to , about debug It is useful to , That's it suspend=y,y Indicates that the JVM Will pause waiting , until IDEA The debugger is connected to continue execution . If you set suspend=n, Then the remote service will not wait IDEA Connect .

In addition, there is another point to pay attention to , Because in the last step, we are remote debug It specifies host and port, So here we must Don't configure deploy-mode by cluster Cluster pattern . because The cluster mode is to start an application on a random machine , It is very likely that it will not start on the server we are listening to , So remote debug The deployment mode of is usually client（ When we do not configure the deployment mode, the default is client）