Spark calculation operator and some small details in liunx

Spark -map operator

map operator ：

object Spark01_Oper {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("Value")
    val cs = new SparkContext(conf)

    val make = cs.makeRDD(1 to 10)
    //map operator 
    val mapRdd = make.map(x => x * 2)
    mapRdd.collect().foreach(println)
  }
}

map Operators are used to calculate the data in all incoming partitions one by one .

mapPartRdd operator

object Spark02_OPer {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("mapPart")
    val sc = new SparkContext(conf)

    //map operator 
    val list = sc.makeRDD(1 to 10)
    val mapPartRdd = list.mapPartitions(datas => {datas.map(data => data*2)})
    mapPartRdd.collect().foreach(println)
  }

}

mapPartRdd operator Be similar to map But it calculates data by partition , The output value of his calculation is a list

mapPartitionsWithIndex operator

object Spark03_OPer {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("With")
    val sc = new SparkContext(conf)
    val list = sc.makeRDD(1 to 10,2)
    val indexRDD = list.mapPartitionsWithIndex {
      case (num, datas) => {
        datas.map((_," Zone number :"+num))
      }
    }
    indexRDD.collect().foreach(println)
  }
}

mapPartitionsWithIndex The operator is similar to mapPartiyions But in func There will be an index value representing the partition , therefore func There will be one more function similar to Int.

spark Possible problems ：

Because every time the calculation data will produce new data, but it will not be deleted , Accumulating all the time will cause memory overflow （OOM）

Driver And Executor The difference between

Driver：

Driver Just create Spark The classes of context objects can be said to be Driver,Driver yes Spark in Application That is, the code release program , It can be understood that it is written for us spark The main program of the code , Secondly, he is also responsible for Executor To allocate tasks ,Driver There can only be one

Executor:

Executor yes Spark In charge of resource calculation , He can exist in multiple .

difference ：

Drvier Like a boss , and Executor yes Driver The hands of ,Driver Be responsible for assigning tasks to Executor To execute .

Linux Pick up

linux Method of switching on and off

1. To turn it off ： shutdown -h restart ：shutdown -r

2. To turn it off ： inti -0 restart ： init -6

3. To turn it off : poweroff restart :reboot

service And systemctl The difference between

service：

You can start 、 stop it 、 Restart and shut down system services , It can also display the current status of all system services ,service The function of the command is to /etc/init.d Find the corresponding service under the directory , Open and close

systemctl：

It's a systemd Tools , Mainly responsible for control systemd System and service manager , yes service and chkconfig The combination of orders

Operation of network equipment ：

Environment variable loading order