Spark SQL chasing Wife Series (initial understanding)

Today is another day , Another day like you

Small talk

         I haven't written an article for a long time , Although four or five days have passed , Writing an article today is very simple , Itchy hands are trying to write an article . Today's article is about Spark SQL Series of articles , About the last wife chasing series RDD Programming , Didn't give up , I will still write . The basic action operator and transformation operator have been written . Then there are accumulators, partitions, and other data types .

         Today's update is Spark SQL, Why not write RDD Well , It's simple ,SQL To write well , Look for a job

Spark SQL What is it?

         If nothing happens ,Spark SQL Will appear in the blog for a long time .

         Let's first introduce what is Spark SQL Well

        Spark SQl The predecessor was Shark, at that time Hadoop There are Hive, It can be used Hsql To replace mr Program to complete data analysis , Very convenient , The difficulty of development is greatly reduced , At that time Spark The ecosystem does not , So the predecessor was created Shark, later Shark No maintenance , Comprehensive in Spark Above to achieve Shark. That's what we have now Spark SQL.

         Although I say Spark It has its own ecosystem , however Spark Most of it is in Hdfs above . As I said before ,MR Out of date , however hdfs This storage system is not out of date ,Spark It's just hdfs What's going on above .

Spark SQL Can do

         After all, SQL Boy, Then from ETL To explain Spark SQL What can I do

1. extract (Extract):Spark SQL From the file system (HDFS, The local system ), Get data from relational database or non relational data .Spark SQL The supported file types are csv,json,xml,Parquet,ORC,Avro etc. .
2. transformation (transform)： It is called data cleaning
3. load (load): The processed data can be stored in different data sources .

        Spark It is mainly used to deal with structured data , What is structured data ？

         Structured data refers to a data set in which the record content has clear structural information and each record in the data set conforms to the structural specification , A data set logically expressed and implemented by a two-dimensional table structure . for instance , It refers to the fields, attributes, types and other information of the relational database .

Spark SQL Key points of

        Data Frame. Can pass Data Frame Of API To analyze the data , be called DSL.

         meanwhile , You can also put Data Frame Registration form , And then use SQL perhaps Hive SQL To do data analysis . For skilled use SQL We can get started more quickly

         Now that we're here Data Frame, So let's introduce

         Learn before RDD When ,RDD It is a collection of data , But I don't know what each piece of data is ,Data Frame Not so ,Data Frame It clearly stipulates that each piece of data consists of several named fields . For image comparison , To see pictures

        RDD Only know what is stored Person Object of type ,Data Frame What's stored is each Person Object information .Data Frame Naked swimming ,RDD It's like wearing clothes . A glance , I can't see anything

Spark Session

        Spark SQL The starting point of programming is Spark Session, Just beginning to learn Spark Core When , The entrance of programming is spark conf. Now? Spark SQL The entrance has changed .

        Spark Session You can create Data Frame object , You can read external files and pass SQL Perform query analysis .

         spark conf: Create context objects

        spark Session: establish Spark Session object

establish Data Frame

         By reading the json File to create df object . Have a look first json file

         Start performing

         Read people.json File to create Data Frame object

val sparkConf = new SparkConf().setMaster("local[*]").setAppName("Spark_Sql") 
val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate() 
import sparkSession.implicits._ 
// Read json file  
val frame = sparkSession.read.json("date/people.json")

         How to display the data of this file ？

        frame.show()

         You can see , And here it is , Just like the result of table query

summary

         It will be updated tomorrow Spark SQL Of DSL and Sql Statement .

         Update here today

         I hope my subject 4 can be booked successfully