Operator operation list of spark

Action operation
1、 collect() , The return value is an array , return dataframe Collect all rows
2、 collectAsList() The return value is one java An array of types , return dataframe Collect all rows
3、 count() Return to one number Type of , return dataframe Number of rows in the collection
4、 describe(cols: String*) Returns a mathematically calculated class table value (count, mean, stddev, min, and max), This can pass multiple parameters , Separated by commas , If any field is empty , So don't participate in the operation , Only this pair of numeric fields . for example df.describe("age", "height").show()
5、 first() Go back to the first line , The type is row type
6、 head() Go back to the first line , The type is row type
7、 head(n:Int) return n That's ok , The type is row type
8、 show() return dataframe The value of the set The default is 20 That's ok , The return type is unit
9、 show(n:Int) return n That's ok ,, The return value type is unit
10、 table(n:Int) return n That's ok , The type is row type

dataframe Basic operation
1、 cache() Memory for synchronizing data
2、 columns Return to one string An array of types , The return value is the name of all columns
3、 dtypes Return to one string Two dimensional array of type , The return value is the name and type of all columns
4、 explan() Print the execution plan Physical
5、 explain(n:Boolean) The input value is false perhaps true , The return value is unit The default is false , If input true Will print Logical and physical
6、 isLocal The return value is Boolean type , If the allowed mode is local return true Otherwise return to false
7、 persist(newlevel:StorageLevel) Return to one dataframe.this.type Enter the storage model type
8、 printSchema() Print out the field name and type Print in a tree structure
9、 registerTempTable(tablename:String) return Unit , take df The objects of are only placed in one table , This table is deleted with the deletion of objects
10、 schema return structType type , Return the field name and type according to the structure type
11、 toDF() Back to a new dataframe Type of
12、 toDF(colnames：String*) Return several fields in the parameter to a new dataframe Type of ,
13、 unpersist() return dataframe.this.type type , Remove the data in the pattern
14、 unpersist(blocking:Boolean) return dataframe.this.type type true and unpersist It's the same thing false It's the removal of RDD

Integrated query ：
1、 agg(expers:column*) return dataframe type , Same as mathematical calculation and evaluation
df.agg(max("age"), avg("salary"))
df.groupBy().agg(max("age"), avg("salary"))
2、 agg(exprs: Map[String, String]) return dataframe type , Same as mathematical calculation and evaluation map Type of
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
3、 agg(aggExpr: (String, String), aggExprs: (String, String)*) return dataframe type , Same as mathematical calculation and evaluation
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
4、 apply(colName: String) return column type , Capture the objects entered into the column
5、 as(alias: String) Back to a new dataframe type , It's the original alias
6、 col(colName: String) return column type , Capture the objects entered into the column
7、 cube(col1: String, cols: String*) Return to one GroupedData type , Summarize according to certain fields
8、 distinct duplicate removal Return to one dataframe type
9、 drop(col: Column) Delete a column return dataframe type
10、 dropDuplicates(colNames: Array[String]) Delete the same column Return to one dataframe
11、 except(other: DataFrame) Return to one dataframe, Returns what exists in the current collection but does not exist in other collections
12、 explode[A, B](inputColumn: String, outputColumn: String)(f: (A) ⇒ TraversableOnce[B])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[B]) The return value is dataframe type , This Split a field into more rows
df.explode("name","names") {name :String=> name.split(" ")}.show();
take name Fields are split by spaces , The split fields are placed in names Inside
13、 filter(conditionExpr: String): Brush some data , return dataframe type df.filter("age>10").show(); df.filter(df("age")>10).show(); df.where(df("age")>10).show(); Fine
14、 groupBy(col1: String, cols: String*) Summarize and return according to a write field groupedate type df.groupBy("age").agg(Map("age" ->"count")).show();df.groupBy("age").avg().show(); Fine
15、 intersect(other: DataFrame) Return to one dataframe, stay 2 individual dataframe The elements that exist
16、 join(right: DataFrame, joinExprs: Column, joinType: String)
One is related dataframe, The second condition of Association , The third type of Association ：inner, outer, left_outer, right_outer, leftsemi
df.join(ds,df("name")===ds("name") and df("age")===ds("age"),"outer").show();
17、 limit(n: Int) return dataframe type Go to n Data out
18、 na: DataFrameNaFunctions , You can call dataframenafunctions Filter the function area of df.na.drop().show(); Delete empty rows
19、 orderBy(sortExprs: Column*) do alise Sort
20、 select(cols:string*) dataframe Swipe the fields df.select($"colA", $"colB" + 1)
21、 selectExpr(exprs: String*) Swipe the fields df.selectExpr("name","name as names","upper(name)","age+1").show();
22、 sort(sortExprs: Column*) Sort df.sort(df("age").desc).show(); The default is asc
23、 unionAll(other:Dataframe) Merge df.unionAll(ds).show();
24、 withColumnRenamed(existingName: String, newName: String) Modify the list df.withColumnRenamed("name","names").show();
25、 withColumn(colName: String, col: Column) Add a row df.withColumn("aa",df("name")).show();

spark Calculate two DataFrame The difference between the set 、 intersection 、 Collection - Algorithmic network