Sparksql of spark

1、SparkSQL Introduce

Hive yes shark The forerunner of ,Shark yes SparkSQL The forerunner of ,SparkSQL The root cause is to completely break away from Hive The limitation of .

2、SparkSQL establish DataFrame The way

In the use of SparkSQL when Scala2.0+ How to create the version of
val session: SparkSession = SparkSession
.builder()
.appName(“test”)
.enableHiveSupport()
.getOrCreate()
session.sparkContext.setLogLevel(“Error”)

2.1、 Read json Format file creation DataFrame
session.read.format(“json”).load("./data/json")

2.2、 adopt json Format RDD establish DataFrame
Implicit conversion is required when reading , Import before reading import session.imlicits._
json Data to be converted to DataSet Format data

2.3、 Not json Format RDD establish DataFrame
By way of reflection will not be json Format RDD convert to DataFrame（ Not recommended ）

2.4、 Dynamically create Schema Will not json Format RDD convert to DataFrame

2.5、 Read parquet File creation DataFrame

Read parquet When you file , First read json file , And then to parquet Write to , And then to session.read.format(“parquet”).load("./data/parquet")

Read json Format data and non json Formatted data , To convert data into DataSet The format of ,DataFrame yes DataSet Of Row edition

3、Spark on hive

3.1、 To configure
In your spark Of client Terminal spark Of conf Under the table of contents , establish hive-site.xml File writes the following configuration , Note that nodes are also hive Server side


hive.metastore.uris
thrift://mynode1:9083

4、SparkSQL UDF And UDAF The difference between

UDF: User defined functions .
You can customize the class to implement UDFX Interface .
UDAF: User defined aggregate function .
Realization UDAF If a function wants to customize a class, it must inherit
UserDefinedAggregateFunction class

Open the window function
row_number over(partition x1 order by x2) For the data in the table X1 grouping , according to X2 Sort , Label the data in each group , The label starts from... Within each group 1 Start , The non label within each group is continuous
rank（）over(partition x1 order by x2) For the data in the table X1 grouping , according to X2 Sort , Label the data in each group , The label starts from... Within each group 1 Start , The non label within each group is discontinuous
dense_rank() over (partition x1 order by x2) For the data in the table X1 grouping , according to X2 Sort , Label the data in each group , The label starts from... Within each group 1 Start , The labels in each group are continuous and the same data labels are the same