1 Introduction to spark Foundation

1 Spark What is it?

2 spark Ten years of wind and rain

3 Extended data

4 Spark Four characteristics

5 Spark Framework module

1.6 Spark The mode of operation of

1.7 Spark The role Architecture

8 summary

1 Spark What problem to solve ？

Massive data computing , Offline batch processing and real-time stream calculation can be carried out

2 Spark What are the modules ？

The core SparkCore、SQL Calculation （SparkSQL）、 Flow calculation （SparkStreaming
）、 Figure calculation （GraphX）、 machine learning （MLlib）

3 Spark What are the characteristics ？

Fast 、 Easy to use 、 Strong commonality 、 Multiple modes of operation

4 Hadoop Process based computing and Spark Advantages and disadvantages of thread based approach ？
Hadoop Medium MR Each of them map/reduce task It's all one java Run in process mode , The advantage is that processes are independent of each other , Every task Exclusive process resources , no
There is mutual interference , Easy to monitor , But the problem is task Inconvenient to share data between , The execution efficiency is relatively low . Like multiple map task Reading different data source files requires adding
To each map task in , Cause repeated loading and waste of memory . Thread based computing is to share data and improve execution efficiency ,Spark The minimum execution of threads is adopted
Company , But the disadvantage is that there will be resource competition between threads

5 Spark The mode of operation of ？
Local mode
• Cluster pattern （StandAlone、YARN、K8S）
• Cloud model
6 Spark Operation role of （ contrast YARN）？
• Master： Cluster resource management （ similar ResourceManager）
• Worker： Stand alone resource management （ similar NodeManager）
• Driver： Single task manager （ similar ApplicationMaster）
• Executor： Single task performer （ similar YARN Inside the container Task）

Basic concepts of threads
Thread is CPU The basic dispatching unit of
 A process usually contains multiple threads , Multiple threads under a process share the resources of the process
 Threads between different processes are not visible to each other
 Threads cannot execute independently
 One thread can create and undo another thread