当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-25 15:15:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
- Overview of JS synchronous, asynchronous, macro task and micro task
- Browser workflow (Simplified)
- Simulate setinterval timer with setTimeout
- Sublimetext-win10 cursor following problem
- The implementation process of inheritance and the difference between Es5 and ES6 implementation
- String type time comparison method with error string.compareto
- Image cropper example
- 流程控制(上)
- Stored procedure bias of SQL to LINQ
- 反射-笔记
猜你喜欢

Award winning interaction | 7.19 database upgrade plan practical Summit: industry leaders gather, why do they come?

"How to use" decorator mode

Docker上运行redis以配置文件方式启动,连接客户端报错Error: Server closed the connection

Spark AQE

Understanding the execution order of T-SQL query from the execution order of join on and where

流程控制(上)

Debounce and throttle

Image cropper example

如何解决Visual Stuido2019 30天体验期过后的登陆问题

System.AccessViolationException: 尝试读取或写入受保护的内存。这通常指示其他内存已损坏
随机推荐
redis淘汰策列
Promise object and macro task, micro task
Spark获取DataFrame中列的方式--col,$,column,apply
Scala110-combineByKey
node学习
oracle_12505错误解决方法
C#,C/S升级更新
Raft of distributed consistency protocol
防抖(debounce)和节流(throttle)
spark分区算子partitionBy、coalesce、repartition
Solve the error caused by too large file when uploading file by asp.net
延迟加载源码剖析:
pl/sql 创建并执行oralce存储过程,并返回结果集
Client error: invalid param endpoint is blank
Use the command to check the WiFi connection password under win10 system
流程控制(上)
Pl/sql creates and executes ORALCE stored procedures and returns the result set
Outline and box shadow to achieve the highlight effect of contour fillet
Meanshift clustering-01 principle analysis
Boosting之GBDT源码分析