当前位置:网站首页>Spark DF adds a column
Spark DF adds a column
2022-07-06 00:28:00 【The south wind knows what I mean】
List of articles
- Method 1 : utilize createDataFrame Method , The process of adding new columns is included in building rdd and schema in
- Method 2 : utilize withColumn Method , The process of adding new columns is included in udf Function
- Method 3 : utilize SQL Code , The process of adding new columns is written directly to SQL In the code
- Method four : The above three are to add a judged column , If you want to add a unique sequence number , have access to monotonically_increasing_id
Method 1 : utilize createDataFrame Method , The process of adding new columns is included in building rdd and schema in
val trdd = input.select(targetColumns).rdd.map(x=>{
if (x.get(0).toString().toDouble > critValueR || x.get(0).toString().toDouble < critValueL)
Row(x.get(0).toString().toDouble,"F")
else Row(x.get(0).toString().toDouble,"T")
})
val schema = input.select(targetColumns).schema.add("flag", StringType, true)
val sample3 = ss.createDataFrame(trdd, schema).distinct().withColumnRenamed(targetColumns, "idx")
Method 2 : utilize withColumn Method , The process of adding new columns is included in udf Function
val code :(Int => String) = (arg: Int) => {
if (arg > critValueR || arg < critValueL) "F" else "T"}
val addCol = udf(code)
val sample3 = input.select(targetColumns).withColumn("flag", addCol(input(targetColumns)))
.withColumnRenamed(targetColumns, "idx")
Method 3 : utilize SQL Code , The process of adding new columns is written directly to SQL In the code
input.select(targetColumns).createOrReplaceTempView("tmp")
val sample3 = ss.sqlContext.sql("select distinct "+targetColname+
" as idx,case when "+targetColname+">"+critValueR+" then 'F'"+
" when "+targetColname+"<"+critValueL+" then 'F' else 'T' end as flag from tmp")
Method four : The above three are to add a judged column , If you want to add a unique sequence number , have access to monotonically_increasing_id
// Add sequence number column add a column method 4
import org.apache.spark.sql.functions.monotonically_increasing_id
val inputnew = input.withColumn("idx", monotonically_increasing_id)
边栏推荐
- MySQL存储引擎
- Introduction of motor
- FFMPEG关键结构体——AVFormatContext
- Notepad++ regular expression replacement string
- Tools to improve work efficiency: the idea of SQL batch generation tools
- State mode design procedure: Heroes in the game can rest, defend, attack normally and attack skills according to different physical strength values.
- 【NOI模拟赛】Anaid 的树(莫比乌斯反演,指数型生成函数,埃氏筛,虚树)
- FFmpeg学习——核心模块
- Huawei equipment is configured with OSPF and BFD linkage
- Huawei equipment configuration ospf-bgp linkage
猜你喜欢
FFmpeg学习——核心模块
The relationship between FPGA internal hardware structure and code
Gavin teacher's perception of transformer live class - rasa project actual combat e-commerce retail customer service intelligent business dialogue robot system behavior analysis and project summary (4
XML Configuration File
Ffmpeg learning - core module
Teach you to run uni app with simulator on hbuilderx, conscience teaching!!!
Start from the bottom structure and learn the introduction of fpga---fifo IP core and its key parameters
Location based mobile terminal network video exploration app system documents + foreign language translation and original text + guidance records (8 weeks) + PPT + review + project source code
[designmode] composite mode
[noi simulation] Anaid's tree (Mobius inversion, exponential generating function, Ehrlich sieve, virtual tree)
随机推荐
Hardware and interface learning summary
Transport layer protocol ----- UDP protocol
2022-02-13 work record -- PHP parsing rich text
DEJA_VU3D - Cesium功能集 之 055-国内外各厂商地图服务地址汇总说明
选择致敬持续奋斗背后的精神——对话威尔价值观【第四期】
【EI会议分享】2022年第三届智能制造与自动化前沿国际会议(CFIMA 2022)
MySQL storage engine
MySql——CRUD
OS i/o devices and device controllers
[designmode] composite mode
Spark SQL空值Null,NaN判断和处理
Spark获取DataFrame中列的方式--col,$,column,apply
Arduino六足机器人
MySQL global lock and table lock
FFMPEG关键结构体——AVCodecContext
Huawei equipment is configured with OSPF and BFD linkage
Detailed explanation of APP functions of door-to-door appointment service
DEJA_ Vu3d - cesium feature set 055 - summary description of map service addresses of domestic and foreign manufacturers
XML Configuration File
Gavin teacher's perception of transformer live class - rasa project actual combat e-commerce retail customer service intelligent business dialogue robot system behavior analysis and project summary (4