Spark overview

Spark What is it?

Spark It is a memory based fast 、 Universal 、 Scalable big data analysis and calculation engine .

Spark and Hadoop

Hadoop By java language-written , Store massive data on the distributed server cluster and run the distributed server Open source framework for analytical applications .

Spark It's a kind of Scala Fast language development 、 Universal 、 Scalable big data analysis engine . The main function is mainly used for data calculation .

Spark or Hadoop

Hadoop MapReduce Because its original design is not to meet the circular iterative data flow processing , So in many Data reusability scenarios running in parallel （ Such as ： machine learning 、 Graph mining algorithm 、 Interactive data mining algorithm ） Zhongcun In many problems such as computational efficiency . therefore Spark emerge as the times require ,Spark It's in the traditional MapReduce Calculation box On the basis of the frame , Using the optimization of its calculation process , Computing based on memory , To reduce the IO The cost of .

Spark and Hadoop The fundamental difference is the problem of data communication between multiple jobs : Spark Data between multiple jobs Communication is based on memory , and Hadoop It's disk based .
Spark Only in shuffle Write data to disk when , and Hadoop More than one of them MR Data interaction between jobs depends on disk interaction ,Spark The caching mechanism is better than HDFS Efficient caching mechanism .

In most data computing scenarios ,Spark It does. MapReduce Have more advantages . however Spark It's memory based , So in the actual production environment , Due to memory limitations , May be Due to insufficient memory resources Job Execution failure , here ,MapReduce It's actually a better choice , therefore Spark It's not a complete replacement for MR.

Spark Core module

• Spark Core Spark Core Provided in Spark The most basic and core functions ,Spark Other functions such as ：Spark SQL, Spark Streaming,GraphX, MLlib It's all in Spark Core On the basis of
• Spark SQL Spark SQL yes Spark Components used to manipulate structured data . adopt Spark SQL, Users can use SQL perhaps Apache Hive Version of SQL dialect （HQL） To query data .
• Spark Streaming Spark Streaming yes Spark Platform for real-time data stream computing components , Provides a wealth of processing Streaming API.
• Spark MLlib MLlib yes Spark Provides a library of machine learning algorithms .MLlib Not only does it provide model evaluation 、 Data import, etc Extra features , It also provides some lower level machine learning primitives .
• Spark GraphX GraphX yes Spark The framework and algorithm library provided by graph computing .