Introduction to partition operators, broadcast variables and accumulators of 32 spark

17.11 operator ( Partition )️

17.11.1 Conversion operator

1. mapPartitionsWithIndex

   • Be similar to mapPartitions, In addition, it also carries the index value of the partition

2. repartition

   • Add or subtract partitions . This operator produces shuffle

3. coalesce

   • coalesce Often used to reduce partitions , The second parameter in the operator is whether the partition reduction process produces shuffle .
   • true In order to produce shuffle , false Do not produce shuffle . The default is false .
   • If coalesce Set the number of partitions than the original RDD If the number of partitions is more , The second parameter is set to false It won't work （ The number of partitions after conversion is greater than that before ）, If I set it to true , Effect and repartition equally .

   repartition(numPartitions) = coalesce(numPartitions,true)
   

4. groupByKey

   • It works on K,V Format RDD On . according to Key Grouping . It works on (K,V) , return (KIterable<V>)
   • groupByKey and reduceByKey difference
     • reduceByKey Is a grouping aggregation class operator , stay Map The end turns on aggregation by default , And the aggregation logic must be consistent with Reduce End consistent , That is, the aggregate function passed in by f Appoint ;
     • groupByKey Is a grouping collection class operator , stay Map End will not produce combine() , Just put the same key The data are collected together , Will not receive similar f Function parameters of

5. zip

   • Put two RDD The elements in （ KV Format / Not KV Format ） To become a KV Format RDD , Two RDD The number of must be the same , At the same time, the number of partitions must also be the same .

6. zipWithIndex

   • This function will RDD The element in and the element in RDD Index number in （ from 0 Start ） Combine into （K,V） Yes

17.11.2 Action operator

1. countByKey
   • Effect to K,V Format RDD On , according to Key The count is the same Key Data set elements of .
2. countByValue
   • Count according to the same content of each element in the dataset . Returns the number of elements with the same content .
3. reduce
   • Aggregate each element in the dataset according to the aggregation logic

17.12 Case answer ️

17.12.1 PV&UV

17.12.2 Two order

17.12.3 Take... In groups topN

17.13 Broadcast variables and accumulators ️

17.13.1 Broadcast variables

The illustration ：

1. The use of broadcast variables

   val conf = new SparkConf()
   conf.setMaster("local").setAppName("brocast")
   val sc = new SparkContext(conf)
   val list = List("hello yjx")
   val broadCast = sc.broadcast(list)
   val lineRDD = sc.textFile("./words.txt")
   lineRDD.filter {
          x => broadCast.value.contains(x) }.foreach {
         println}
   sc.stop()
   

2. matters needing attention

   • Broadcast variables can only be Driver End definition , Can't be in Executor End definition .
   • stay Driver You can change the value of the broadcast variable , stay Executor End can't modify the value of broadcast variable

17.13.2 accumulator

The illustration ：

1. The use of accumulators

   val conf = new SparkConf()
   conf.setMaster("local").setAppName("accumulator")
   val sc = new SparkContext(conf)
   val accumulator = sc.longAccumulator
   sc.textFile("./words.txt").foreach {
          x =>{
         accumulator.add(1)}}
   println(accumulator.value)
   sc.stop()
   

2. matters needing attention

   • Accumulator in Driver Initial value assigned to end definition , Accumulator can only be Driver End read , stay Executor End update