当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-06 00:28:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
- LeetCode 8. String conversion integer (ATOI)
- Yolov5、Pycharm、Anaconda环境安装
- MySQL global lock and table lock
- 【NOI模拟赛】Anaid 的树(莫比乌斯反演,指数型生成函数,埃氏筛,虚树)
- [noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
- Hudi of data Lake (2): Hudi compilation
- 【EI会议分享】2022年第三届智能制造与自动化前沿国际会议(CFIMA 2022)
- Key structure of ffmpeg - avformatcontext
- Hardware and interface learning summary
- Codeforces gr19 D (think more about why the first-hand value range is 100, JLS yyds)
猜你喜欢
Leetcode 450 deleting nodes in a binary search tree
Detailed explanation of APP functions of door-to-door appointment service
Extension and application of timestamp
FFmpeg抓取RTSP图像进行图像分析
免费的聊天机器人API
MySql——CRUD
Common API classes and exception systems
anconda下载+添加清华+tensorflow 安装+No module named ‘tensorflow‘+KernelRestarter: restart failed,内核重启失败
Priority queue (heap)
权限问题:source .bash_profile permission denied
随机推荐
Ffmpeg learning - core module
FFMPEG关键结构体——AVFormatContext
Data analysis thinking analysis methods and business knowledge -- analysis methods (II)
Leetcode 450 deleting nodes in a binary search tree
7.5模拟赛总结
Date类中日期转成指定字符串出现的问题及解决方法
Go learning --- read INI file
如何制作自己的机器人
Hardware and interface learning summary
FFT learning notes (I think it is detailed)
Search (DFS and BFS)
建立时间和保持时间的模型分析
[designmode] adapter pattern
Room cannot create an SQLite connection to verify the queries
LeetCode 6005. The minimum operand to make an array an alternating array
Extracting profile data from profile measurement
Shardingsphere source code analysis
7.5 decorator
数据分析思维分析方法和业务知识——分析方法(三)
Browser local storage