当前位置:网站首页>How spark gets columns in dataframe --column, $, column, apply
How spark gets columns in dataframe --column, $, column, apply
2022-07-06 00:28:00 【The south wind knows what I mean】
Spark obtain DataFrame The way listed in --col,$,column,apply
1 Official statement
df("columnName") // On a specific data frame
col("columnName") // Generic columns that have not been associated with data frames
col("columnName.field") // Extract structural fields
col("`a.column.with.dots`") // escape . In the column name
$"columnName" //Scala Abbreviations of named columns
expr("a + 1") // Resolved by SQL Expression constructed column
lit("abc") // Generate text ( Constant ) Columns of values
2 Packages involved in use
//
import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
3 Demo
//
scala> val idCol = $"id"
idCol: org.apache.spark.sql.ColumnName = id
scala> val idCol = col("id")
idCol: org.apache.spark.sql.Column = id
scala> val idCol = column("id")
idCol: org.apache.spark.sql.Column = id
scala> val dataset = spark.range(5).toDF("text")
dataset: org.apache.spark.sql.DataFrame = [text: bigint]
scala> val textCol = dataset.col("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset.apply("text")
textCol: org.apache.spark.sql.Column = text
scala> val textCol = dataset("text")
textCol: org.apache.spark.sql.Column = text
边栏推荐
- Opencv classic 100 questions
- 时间戳的拓展及应用实例
- Start from the bottom structure and learn the introduction of fpga---fifo IP core and its key parameters
- 云导DNS和知识科普以及课堂笔记
- Mysql - CRUD
- Search (DFS and BFS)
- AtCoder Beginner Contest 254【VP记录】
- LeetCode 6004. Get operands of 0
- LeetCode 6006. Take out the least number of magic beans
- Leetcode Fibonacci sequence
猜你喜欢
Ffmpeg learning - core module
Location based mobile terminal network video exploration app system documents + foreign language translation and original text + guidance records (8 weeks) + PPT + review + project source code
MySql——CRUD
Determinant learning notes (I)
Multithreading and high concurrency (8) -- summarize AQS shared lock from countdownlatch (punch in for the third anniversary)
Go learning - dependency injection
wx. Getlocation (object object) application method, latest version
认识提取与显示梅尔谱图的小实验(观察不同y_axis和x_axis的区别)
建立时间和保持时间的模型分析
剖面测量之提取剖面数据
随机推荐
[designmode] Decorator Pattern
Key structure of ffmpeg - avframe
Mysql - CRUD
Data analysis thinking analysis methods and business knowledge - analysis methods (III)
LeetCode 6006. Take out the least number of magic beans
Solve the problem of reading Chinese garbled code in sqlserver connection database
Intranet Security Learning (V) -- domain horizontal: SPN & RDP & Cobalt strike
如何利用Flutter框架开发运行小程序
[binary search tree] add, delete, modify and query function code implementation
Opencv classic 100 questions
Global and Chinese markets of universal milling machines 2022-2028: Research Report on technology, participants, trends, market size and share
《编程之美》读书笔记
Notepad++ regular expression replacement string
【线上小工具】开发过程中会用到的线上小工具合集
Priority queue (heap)
如何解决ecology9.0执行导入流程流程产生的问题
MySQL global lock and table lock
Hardware and interface learning summary
Reading notes of the beauty of programming
The global and Chinese markets of dial indicator calipers 2022-2028: Research Report on technology, participants, trends, market size and share