当前位置:网站首页>Spark SQL UDF function
Spark SQL UDF function
2022-07-06 00:28:00 【The south wind knows what I mean】
UDF function
def main(args: Array[String]): Unit = {
val session: SparkSession = SparkSession
.builder()
.config("dfs.ha.namenodes.nameservices","xxxxx")
.appName("sql demo")
.master("local")
.enableHiveSupport()
.getOrCreate()
// Custom function
val code: UserDefinedFunction = session.udf.register("code", (arg: Int) => {
if (arg > 5) "F" else "M"
})
val sql_1:String =
s""" |select * from |sparktuning.course_pay |""".stripMargin
val df: DataFrame = session.sql(sql_1)
// usage 1.
df.select("chapterid")
.withColumn("flag",code(col("chapterid")))
.withColumnRenamed("chapterid","new_chapterid")
.show(false)
+-------------+----+
|new_chapterid|flag|
+-------------+----+
|4 |M |
|7 |F |
|8 |F |
|5 |M |
|4 |M |
|9 |F |
|7 |F |
|0 |M |
|5 |M |
|4 |M |
|4 |M |
|0 |M |
|1 |M |
|4 |M |
|9 |F |
+-------------+----+
// usage 2.
val sql_2:String =
s""" |select | code(chapterid) Sex |from |sparktuning.course_pay |""".stripMargin
session.sql(sql_2).show(false)
+---+
|Sex|
+---+
|M |
|F |
|F |
|M |
|M |
|F |
|F |
|M |
|M |
|M |
|M |
|M |
|M |
|M |
|F |
+---+
}
边栏推荐
- Power Query数据格式的转换、拆分合并提取、删除重复项、删除错误、转置与反转、透视和逆透视
- STM32 configuration after chip replacement and possible errors
- About the slmgr command
- Problems and solutions of converting date into specified string in date class
- MySQL global lock and table lock
- The relationship between FPGA internal hardware structure and code
- Atcoder beginer contest 258 [competition record]
- Yolov5、Pycharm、Anaconda环境安装
- What is information security? What is included? What is the difference with network security?
- N1 # if you work on a metauniverse product [metauniverse · interdisciplinary] Season 2 S2
猜你喜欢
Huawei equipment is configured with OSPF and BFD linkage
关于slmgr命令的那些事
Spark AQE
认识提取与显示梅尔谱图的小实验(观察不同y_axis和x_axis的区别)
FFMPEG关键结构体——AVFormatContext
notepad++正則錶達式替換字符串
How to use the flutter framework to develop and run small programs
Hudi of data Lake (1): introduction to Hudi
Priority queue (heap)
Idea远程提交spark任务到yarn集群
随机推荐
Choose to pay tribute to the spirit behind continuous struggle -- Dialogue will values [Issue 4]
7.5 simulation summary
[designmode] composite mode
LeetCode 6006. Take out the least number of magic beans
LeetCode 1598. Folder operation log collector
Data analysis thinking analysis methods and business knowledge - analysis methods (III)
MySQL functions
Problems encountered in the database
MySQL之函数
Global and Chinese market of valve institutions 2022-2028: Research Report on technology, participants, trends, market size and share
MySQL storage engine
JS can really prohibit constant modification this time!
[noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
State mode design procedure: Heroes in the game can rest, defend, attack normally and attack skills according to different physical strength values.
Gd32f4xx UIP protocol stack migration record
Mysql - CRUD
Key structure of ffmpeg - avformatcontext
Reading notes of the beauty of programming
【DesignMode】适配器模式(adapter pattern)
LeetCode 6004. Get operands of 0