当前位置:网站首页>Spark-SQL UDF函数
Spark-SQL UDF函数
2022-07-06 00:23:00 【南风知我意丿】
UDF函数
def main(args: Array[String]): Unit = {
val session: SparkSession = SparkSession
.builder()
.config("dfs.ha.namenodes.nameservices","xxxxx")
.appName("sql demo")
.master("local")
.enableHiveSupport()
.getOrCreate()
//自定义函数
val code: UserDefinedFunction = session.udf.register("code", (arg: Int) => {
if (arg > 5) "F" else "M"
})
val sql_1:String =
s""" |select * from |sparktuning.course_pay |""".stripMargin
val df: DataFrame = session.sql(sql_1)
//用法1.
df.select("chapterid")
.withColumn("flag",code(col("chapterid")))
.withColumnRenamed("chapterid","new_chapterid")
.show(false)
+-------------+----+
|new_chapterid|flag|
+-------------+----+
|4 |M |
|7 |F |
|8 |F |
|5 |M |
|4 |M |
|9 |F |
|7 |F |
|0 |M |
|5 |M |
|4 |M |
|4 |M |
|0 |M |
|1 |M |
|4 |M |
|9 |F |
+-------------+----+
//用法2.
val sql_2:String =
s""" |select | code(chapterid) Sex |from |sparktuning.course_pay |""".stripMargin
session.sql(sql_2).show(false)
+---+
|Sex|
+---+
|M |
|F |
|F |
|M |
|M |
|F |
|F |
|M |
|M |
|M |
|M |
|M |
|M |
|M |
|F |
+---+
}
边栏推荐
- 什么叫做信息安全?包含哪些内容?与网络安全有什么区别?
- 多线程与高并发(8)—— 从CountDownLatch总结AQS共享锁(三周年打卡)
- OpenCV经典100题
- Calculate sha256 value of data or file based on crypto++
- 18. (ArcGIS API for JS) ArcGIS API for JS point collection (sketchviewmodel)
- FFT 学习笔记(自认为详细)
- 【在线聊天】原来微信小程序也能回复Facebook主页消息!
- Gd32f4xx UIP protocol stack migration record
- Global and Chinese market of valve institutions 2022-2028: Research Report on technology, participants, trends, market size and share
- Huawei equipment configuration ospf-bgp linkage
猜你喜欢
Senparc. Weixin. Sample. MP source code analysis
Data analysis thinking analysis methods and business knowledge -- analysis methods (II)
【EI会议分享】2022年第三届智能制造与自动化前沿国际会议(CFIMA 2022)
[noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
MySQL存储引擎
时间戳的拓展及应用实例
【在线聊天】原来微信小程序也能回复Facebook主页消息!
FFMPEG关键结构体——AVFrame
How to solve the problems caused by the import process of ecology9.0
【DesignMode】装饰者模式(Decorator pattern)
随机推荐
Leetcode 450 deleting nodes in a binary search tree
Upgrade openssl-1.1.1p for openssl-1.0.2k
Hudi of data Lake (2): Hudi compilation
Wechat applet -- wxml template syntax (with notes)
Leetcode:20220213 week race (less bugs, top 10% 555)
Recognize the small experiment of extracting and displaying Mel spectrum (observe the difference between different y_axis and x_axis)
Huawei equipment is configured with OSPF and BFD linkage
Location based mobile terminal network video exploration app system documents + foreign language translation and original text + guidance records (8 weeks) + PPT + review + project source code
What are Yunna's fixed asset management systems?
[Chongqing Guangdong education] Chongqing Engineering Vocational and Technical College
There is no network after configuring the agent by capturing packets with Fiddler mobile phones
LeetCode 斐波那契序列
如何解决ecology9.0执行导入流程流程产生的问题
【在线聊天】原来微信小程序也能回复Facebook主页消息!
Data analysis thinking analysis methods and business knowledge -- analysis methods (II)
Teach you to run uni app with simulator on hbuilderx, conscience teaching!!!
Intranet Security Learning (V) -- domain horizontal: SPN & RDP & Cobalt strike
Key structure of ffmpeg - avformatcontext
Global and Chinese markets of universal milling machines 2022-2028: Research Report on technology, participants, trends, market size and share
关于结构体所占内存大小知识