当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-06 00:28:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
- Data analysis thinking analysis methods and business knowledge - analysis methods (III)
- Global and Chinese market of digital serial inverter 2022-2028: Research Report on technology, participants, trends, market size and share
- FFT learning notes (I think it is detailed)
- MySQL存储引擎
- 小程序技术优势与产业互联网相结合的分析
- The global and Chinese markets of dial indicator calipers 2022-2028: Research Report on technology, participants, trends, market size and share
- Yolov5, pychar, Anaconda environment installation
- Permission problem: source bash_ profile permission denied
- 【DesignMode】装饰者模式(Decorator pattern)
- 7.5 simulation summary
猜你喜欢
[noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
Choose to pay tribute to the spirit behind continuous struggle -- Dialogue will values [Issue 4]
如何利用Flutter框架开发运行小程序
Calculate sha256 value of data or file based on crypto++
Hudi of data Lake (2): Hudi compilation
Extracting profile data from profile measurement
选择致敬持续奋斗背后的精神——对话威尔价值观【第四期】
Tools to improve work efficiency: the idea of SQL batch generation tools
【EI会议分享】2022年第三届智能制造与自动化前沿国际会议(CFIMA 2022)
FFMPEG关键结构体——AVCodecContext
随机推荐
Key structure of ffmpeg -- AVCodecContext
Go learning --- structure to map[string]interface{}
[noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
Tools to improve work efficiency: the idea of SQL batch generation tools
【DesignMode】适配器模式(adapter pattern)
Spark DF增加一列
Configuring OSPF GR features for Huawei devices
Leetcode Fibonacci sequence
Data analysis thinking analysis methods and business knowledge - analysis methods (III)
AtCoder Beginner Contest 258【比赛记录】
选择致敬持续奋斗背后的精神——对话威尔价值观【第四期】
电机的简介
Pointer pointer array, array pointer
Global and Chinese markets of POM plastic gears 2022-2028: Research Report on technology, participants, trends, market size and share
从底层结构开始学习FPGA----FIFO IP核及其关键参数介绍
Hardware and interface learning summary
Global and Chinese market of water heater expansion tank 2022-2028: Research Report on technology, participants, trends, market size and share
7.5模拟赛总结
Mysql - CRUD
[designmode] composite mode