Why rdd/dataset is needed in spark

No, RDD、Dataset Before , do wordcount Or other big data computing ：

• Native python aggregate ： such as python Of list、set、map, But only stand-alone version is supported , Distributed... Is not supported . If you want to do Distributed Computing , A lot of extra work needs to be done , Like threads / Process communication , Fault tolerance , Automatic load balancing .. It's troublesome to wait . So the framework was born .

• You can also use MapReduce： Inefficient operation , Development efficiency is also low .

• The birth of Spark/Flink, Reference to native scala The design of the collection , Abstract out new data types -RDD/Dataset

  RDD It's actually a distributed collection , Support function operation .. It is as simple to use as a local collection . Fast development speed . The bottom layer is based on distributed memory computing , yes MR Of 100 times .

• 

   

RDD What is it

• Elastic distributed data sets

• elastic ：【 Memory 】 and 【CPU】 Can be extended . Intermediate data exists 【 Memory 】, If there's not enough memory , Can overflow 【 disk 】

• Distributed ：【 Storage 】 and 【 Calculation 】 Are distributed on multiple nodes

• Data sets ： A large Abstract container , Use it to follow python Collection is as simple , Support 【 Functional expression 】 Programming .

 

 

Core design points

• immutable ： The elements of the set cannot be changed inside , But it can be converted into a new set .

• Divisible ：RDD Divided into several parts .

• Parallel computing ： Each partition is handled by a task , Each task is calculated in parallel .

Recorded in the source code RDD Of 5 Big characteristic

• There is a partition list ： take 【 All data 】 Divide into a reasonable number of partitions .

• Calculation function ： Each partition has 【 function 】

• Dependency list ：RDD Convert to a new RDD, Dependencies are also recorded .

• 【 Optional 】 Comparator ： When RDD The element is 【 Key value pair 】 when , You can specify a partition , Specifies how to press key To group into different partitions . The default is Hash Comparator

• 【 Optional 】 The best position ： The best position of calculation is recorded （ Moving code is more cost-effective than moving data ） such as HDFS Of block Location .

 

 

RDD Of 5 The big feature actually shows ：

Where is the data ？ Where to calculate ？ What are the divisions ？ What partition to use ？ What function is used to calculate ？