当前位置:网站首页>Spark获取DataFrame中列的方式--col,$,column,apply

Spark获取DataFrame中列的方式--col,$,column,apply

2022-07-06 00:23:00 南风知我意丿

Spark获取DataFrame中列的方式--col,$,column,apply

1 官方说明

df("columnName")       		//在特定数据帧上 
col("columnName")           //尚未与数据帧关联的通用列
col("columnName.field")     //提取结构字段
col("`a.column.with.dots`") //转义。在列名中
$"columnName"          		//Scala命名列的缩写 
expr("a + 1")          		//由解析的SQL表达式构造的列 
lit("abc") 					//产生文字(常量)值的列

2 使用时涉及到的的包

//
   import spark.implicits._
   import org.apache.spark.sql.functions._
   import org.apache.spark.sql.Column

3 Demo

//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
 
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
 
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id

scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
 
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
 
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
 
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
原网站

版权声明
本文为[南风知我意丿]所创,转载请带上原文链接,感谢
https://blog.csdn.net/Lzx116/article/details/125603787