The spark operator - repartition operator

Source code

def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {coalesce(numPartitions, shuffle = true)}

1. The bottom layer calls coalesce() to implement the repartitioning operation
2. Mandatory shuffle operation

Repartition and coalesce relationship

1. All operators are used to reset the number of partitions
2. The bottom layer of repartition calls coalesce() to implement the repartition function
3. repartition forces a shuffle operation
4. coalesce decides whether to perform the shuffle operation according to the parameters

References

spark operator-coalesce operator