Sparkshuffle process and Mr shuffle process

• SparkShuffle

  • Spark1.2 In the version 【HashShuffle】

  • Spark1.2 In later versions 【sortShuffle】

• MR Of shuffle

 

 Spark Of shuffle

HashshuffleManager( Abandon )

• Unoptimized HashShuffle, Number of small files in the middle =【 The upstream task Number 】*【 The downstream task Number 】, To many, many

• groupby and join Can trigger shuffle. The figure below shows the original join situation , Simplified to groupby Aggregation

  The optimized HashShuffle, Number of small files in the middle =【Executor The number of 】*【 The downstream task The number of 】, The number has decreased exponentially .

• groupby and join Can trigger shuffle.

sortshuffleManager( choose )

• Common mechanisms ( Need to sort )

 

• • 1- Define the data structure ： If it is reduceByKey This kind of aggregate class shuffle operator , Then I will choose 【Map】 data structure , If it is join such shuffle operator , Then I will choose 【Array】 data structure

  • 2- Requested memory = Current data memory *2- Last memory condition

  • 3- Sort ： Before overflowing to disk file , According to key The existing data in the memory data structure is 【 Sort 】.

  • 4- Overflow disk ： After sorting , Data will be written to disk files in batches .* default batch The number is 10000 strip , in other words , Sorted data , With every batch of 1 Ten thousand pieces of data are written to disk files in batches .*

  • 5- Merge ：* In the file start offset And end offset.* Indicates the file index

• ByPass Mechanism ( There is no need to sort )

  • When shuffle write task Less than or equal to 【spark.shuffle.sort.bypassMergeThreshold】 The value of the parameter ( The default is 【200】)

  • It can't be 【 with map End aggregated shuffle operator 】.

    • reduceByKey yes map End aggregate class shuffle operator .

    • groupBykey No map End aggregate class shuffle operator .