Principle analysis of spark

Spark Principle analysis of

---

Preface

Today's main learning is a Spark Analysis of the principle of the framework ,spark Operation flow ,RDD An execution process of , An introduction to dependencies

---

One . Spark brief introduction

Spark By scala Developed ,scala To run on JAVA platform （JVM）, And compatible with existing JAVA Program , So use scala The program is written by Java jdk You can run , It doesn't need to scala jdk

Spark And MapReduce contrast

As can be seen from the figure above , Use Hadoop MR Iterative computation is very resource intensive
Spark After loading the data into memory , The subsequent iterative calculation can directly use the intermediate results in memory for operation , Avoid frequently reading data from disk

Two . Basic concept and architecture design

Spark The basic process of operation （ Here we use YARN For example ）

1. When the client submits the application , First, build a basic running environment for applications SparkContext And to RM Register and apply for resources , by Driver Create a SC Apply for resources , Assignment and monitoring of tasks
   there Driver Understand applications written for users ,SparkContex（SC）t Is similar to RM Medium AM function
2. RM After receiving the request, it will start Executor And allocate resources , And to SC Register and apply for Task, And always with SC Maintain communication to prevent disconnection .
3. After the job runs SC towards RM Apply for cancellation and close yourself

RDD A basic operational overview of

RDD The typical execution process of is as follows

1.RDD Read in the external data source and create , If the data source is large, multiple partitions will be created , Different partitions will go to different data nodes , Because of this characteristic , Talent RDD: Distributed elastic datasets
2.RDD After a series of conversion operations ： Each conversion operation will form a new RDD For the next conversion operation , In this way, it forms DAG chart
3. the last one RDD Output to external data source through action operation
In the process ,RDD Will convert , But it will not generate specific results , Only encounter action operation （action） Will calculate the corresponding results

RDD Dependency of

As shown in the figure above ,RDD There are wide dependence and narrow dependence , What has narrow dependence , Now let me talk about Kuan dependence , It shows that there is a father RDD One partition of the corresponding sub RDD Multiple sections of

summary

Today's writing is more water , In general, it is written to consolidate what we have learned today . I'll be more specific when I have time later