当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-06 00:28:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
- Huawei equipment configuration ospf-bgp linkage
- Multithreading and high concurrency (8) -- summarize AQS shared lock from countdownlatch (punch in for the third anniversary)
- Leetcode Fibonacci sequence
- 数据分析思维分析方法和业务知识——分析方法(三)
- Ffmpeg learning - core module
- Global and Chinese market of water heater expansion tank 2022-2028: Research Report on technology, participants, trends, market size and share
- Reading notes of the beauty of programming
- Global and Chinese market of valve institutions 2022-2028: Research Report on technology, participants, trends, market size and share
- 如何解决ecology9.0执行导入流程流程产生的问题
- [binary search tree] add, delete, modify and query function code implementation
猜你喜欢
Go learning - dependency injection
[EI conference sharing] the Third International Conference on intelligent manufacturing and automation frontier in 2022 (cfima 2022)
Analysis of the combination of small program technology advantages and industrial Internet
Basic introduction and source code analysis of webrtc threads
Classic CTF topic about FTP protocol
OS i/o devices and device controllers
Spark SQL空值Null,NaN判断和处理
The relationship between FPGA internal hardware structure and code
Hardware and interface learning summary
【DesignMode】装饰者模式(Decorator pattern)
随机推荐
从底层结构开始学习FPGA----FIFO IP核及其关键参数介绍
AtCoder Beginner Contest 258【比赛记录】
Shardingsphere source code analysis
常用API类及异常体系
Gavin teacher's perception of transformer live class - rasa project actual combat e-commerce retail customer service intelligent business dialogue robot system behavior analysis and project summary (4
【DesignMode】装饰者模式(Decorator pattern)
Start from the bottom structure and learn the introduction of fpga---fifo IP core and its key parameters
Wechat applet -- wxml template syntax (with notes)
Key structure of ffmpeg - avformatcontext
Classic CTF topic about FTP protocol
时间戳的拓展及应用实例
Leetcode:20220213 week race (less bugs, top 10% 555)
Room cannot create an SQLite connection to verify the queries
Determinant learning notes (I)
Natural language processing (NLP) - third party Library (Toolkit):allenlp [library for building various NLP models; based on pytorch]
LeetCode 1598. Folder operation log collector
FFmpeg学习——核心模块
MySql——CRUD
DEJA_ Vu3d - cesium feature set 055 - summary description of map service addresses of domestic and foreign manufacturers
Data analysis thinking analysis methods and business knowledge -- analysis methods (II)