Understanding of spark operator aggregatebykey

Yes spark operator aggregateByKey The understanding of the

Case study

aggregateByKey Operators are actually different “key” Data make a map+reduce Operation of the protocol .

Take a simple piece of code in the production environment
There are some sorted log fields , After processing, we got RDD The type is (String,(String,String)) Of List Format results , Each of them String It stands for ：( user name ,( Access time , Access page url))
The same user may visit different or the same pages at different times , In order to merge the access behavior of the same user , Write the following code , be used aggregateByKey.

val data = sc.parallelize(
List(
("13909029812",("20170507","http://www.baidu.com")),("18089376778",("20170401","http://www.google.com")),("18089376778",("20170508","http://www.taobao.com")),("13909029812",("20170507","http://www.51cto.com"))
)
)
        data.aggregateByKey(scala.collection.mutable.Set[(String, String)](), 200)((set, item) => {
          set += item
        }, (set1, set2) => set1 union set2).mapValues(x => x.toIterable).collect

result ：

res12: Array[(String, Iterable[(String, String)])] = Array((18089376778,Set((20170401,http://www.google.com), (20170508,http://www.taobao.com))), (13909029812,Set((20170507,http://www.51cto.com), (20170507,http://www.baidu.com))))

Decomposition analysis ：##

aggregateByKey( Parameters 1)( Parameters 2, Parameters 3)

The process ： about data One of the key, Parameters 1 The initialization value is , In the parameter 2 The function of , Initial value and this key Every one of value Pass in functions to operate , All returned results are in the parameter 3 In .

• Parameters 1

  scala.collection.mutable.Set[(String, String)]()

new An empty set aggregate , As the initial value

• Parameters 2
  (set, item) => {
  set += item
  }
  A similar to map The mapping function of , Will be key Every one of value（ In this case ( Access time , visit url)） As item, Put it in the set And in return .
  Know a key All of the value Will return a containing the value Of set

• Parameters 3
  (set1, set2) => set1 union set2
  The key All of the value Got set Conduct union Statute . And back to

final result ： Get the access of every user at all times url Behavior information .

original text ：https://www.jianshu.com/p/09912beb1350