当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply

How spark gets columns in dataframe --column, $, column, apply

2022-07-06 00:28:00 The south wind knows what I mean

Spark obtain DataFrame The way listed in --col,$,column,apply

1 Official statement

df("columnName")       		// On a specific data frame  
col("columnName")           // Generic columns that have not been associated with data frames 
col("columnName.field")     // Extract structural fields 
col("`a.column.with.dots`") // escape . In the column name 
$"columnName"          		//Scala Abbreviations of named columns  
expr("a + 1")          		// Resolved by SQL Expression constructed column  
lit("abc") 					// Generate text ( Constant ) Columns of values 

2 Packages involved in use

//
   import spark.implicits._
   import org.apache.spark.sql.functions._
   import org.apache.spark.sql.Column

3 Demo

//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
 
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
 
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id

scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
 
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
 
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
 
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
原网站

版权声明
本文为[The south wind knows what I mean]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/187/202207060022444255.html