Aggregate function with key in spark

• RDD Every element of is 【 Key value pair 】 To call the following functions .

• groupByKey

• aggregateByKey

  • rdd = sc.parallelize([('a', 1), ('b', 1), ('a', 1), ('b', 1), ('a', 1)], 2)

  • When aggregating in the following partition , The initial value will participate in the calculation , When aggregating between partitions , The initial value will not participate in the calculation .

 

• foldByKey

  • foldByKey By aggregateByKey Simplify

  • When aggregateByKey The logic of aggregation functions within and between partitions of is the same , It can be omitted as a , It becomes foldByKey.

• reduceByKey

  • reduceByKey By foldByKey Simplify

  • When foldByKey When the initial value of is meaningless , You can omit it