当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-25 15:15:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
猜你喜欢

API health status self inspection

Overview of JS synchronous, asynchronous, macro task and micro task

String type time comparison method with error string.compareto

"Ask every day" reentrantlock locks and unlocks

打开虚拟机时出现VMware Workstation 未能启动 VMware Authorization Service

"How to use" agent mode

Award winning interaction | 7.19 database upgrade plan practical Summit: industry leaders gather, why do they come?

Yarn: the file yarn.ps1 cannot be loaded because running scripts is prohibited on this system.

异步fifo的实现

【JS高级】js之正则相关函数以及正则对象_02
随机推荐
args参数解析
[Android] recyclerview caching mechanism, is it really difficult to understand? What level of cache is it?
RedisCluster搭建和扩容
Application of object detection based on OpenCV and yolov3
浏览器工作流程(简化)
"How to use" decorator mode
6月产品升级观察站
js URLEncode函数
记一次redis超时
图片的懒加载
用setTimeout模拟setInterval定时器
Leetcode combination sum + pruning
Client error: invalid param endpoint is blank
Install entityframework method
bridge-nf-call-ip6tables is an unknown key异常处理
Implementation of asynchronous FIFO
C语言函数复习(传值传址【二分查找】,递归【阶乘,汉诺塔等】)
深入:微任务与宏任务
System.AccessViolationException: 尝试读取或写入受保护的内存。这通常指示其他内存已损坏
一个程序最多可以使用多少内存?