Rational selection of (Spark Tuning ~) operator

1.map And mappartition

1.map It's right RDD Each element in acts on a function

2.mappartition Is to apply a function to each partition

If you need to write the data to the database , Be sure to use mappartition

2.foreach And foreachpartition

Be similar to map And mappartition
The difference is that ：foreach It's an action operator ,map It's a transformation operator

3.groupByKey And reduceByKey

1.groupByKey

All the data has been shuffle.

2.reduceByKey

Will be first map Make a local aggregation at the end , Then aggregate the data shuffle operation （map End prepolymerization ）
（ This method is preferred ）

4.collect operator

The data of the execution result is all in an array of Van Gogh （ It can lead to OOM） Use with caution ！

5.coalesce And repartition

The function of both is to change the number of partitions

1.coalesce operator

When the number of partitions is reduced, there will be no shuffle,（data.coalesce(1)）
When the number of partitions exceeds the default , There will be shuffle

It is generally used in multi partition and less partition

2.repartition operator

repartition Operator bottom call coalesce（shuffle = true）, There will be shuffle