当前位置:网站首页>Use reflection to convert RDD to dataframe
Use reflection to convert RDD to dataframe
2022-07-25 03:01:00 【Rookies also have dreams】
Java:
package cn.spark.sql;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;
import java.util.List;
public class RDD2DataFrameReflection {
public static void main(String[] args){
SparkConf conf = new SparkConf().setAppName("RDD2DataFrameReflection")
.setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
SQLContext sqlContext = new SQLContext(sc);
JavaRDD<String> lines = sc.textFile("C://Users//Desktop//students.txt");
JavaRDD<Student> students = lines.map(
new Function<String, Student>() {
private static final long serialVersionUID = 1L;
@Override
public Student call(String s) throws Exception {
String[] lineSplit = s.split(",");
Student stu = new Student();
stu.setID(Integer.valueOf(lineSplit[0].trim()));
stu.setName(lineSplit[1]);
stu.setAge(Integer.valueOf(lineSplit[2].trim()));
return stu;
}
});
/*
* Use reflection take RDD Convert to DataFrame
* take Student.class Put it in In fact, it is created by reflection DataFrame
* Because the bottom layer passes Student class On reflection To get some of the field
* requirement javabean We need to implement serializable Interface
* */
DataFrame studentDF = sqlContext.createDataFrame(
students,Student.class);
// To get a DataFrame after You can register a temporary table
studentDF.registerTempTable("students");
DataFrame teengerDF = sqlContext.sql("select * from students where age <= 18");
// Will find out DataFrame Convert again to RDD
JavaRDD<Row> teengerRDD = teengerDF.javaRDD();
// take EDD Data in mapping It maps to student
JavaRDD<Student> teengerStudentRDD = teengerRDD.map(new Function<Row, Student>() {
@Override
public Student call(Row row) throws Exception {
Student stu = new Student();
stu.setAge(row.getInt(0));
stu.setID(row.getInt(1));
stu.setName(row.getString(2));
return stu;
}
});
List<Student> studentList = teengerStudentRDD.collect();
for (Student stu : studentList){
System.out.println(stu);
}
}
}
test :

边栏推荐
- "Introduction to interface testing" punch in to learn day05: how can a testing framework support restful interfaces?
- Dc-1-practice
- Pypi counts the number of Downloads
- Conceptual distinction between Po, Bo, VO, dto and POJO
- Dynamic planning of force buckle punch in summary
- It7259q-13, it7259ex-24 feature wearable devices
- Inheritance (prototype)
- FLASH read / write problem of stm32cubemx
- Visualization of correlation coefficient matrix
- Brief understanding of operational amplifier
猜你喜欢

Color space (1) - RGB

Mgre.hdlc.ppp.chap.nat comprehensive experiment

Object.defineproperty use
![[stm32f130rct6] idea and code of ultrasonic ranging module](/img/a6/1bae9d5d8628f00acf4738008a0a01.png)
[stm32f130rct6] idea and code of ultrasonic ranging module

Flink's study notes

Learning record 12

Wechat sports field reservation of the finished works of the applet graduation project (6) opening defense ppt
![[stm32f103rct6] can communication](/img/24/71509bd0d74d43ce4a79b8126478ff.jpg)
[stm32f103rct6] can communication

JS foundation -- object static method

JS foundation -- data
随机推荐
Arduino + si5351 square wave generator
Tp5.1 initialize initialization method (not \u initialize)
JS construction linked list
Domestic edge computing organization and product research
Physical experiment simulation
Sum of "n" numbers of force deduction summary
Use unicloud cloud function to decode wechat motion steps in applet
JS foundation -- object static method
Dc-2-range practice
BGP introduction
SQL Server 2022 installation
hello csdn
[pyGame practice] nostalgic classic - do you remember the name of this chess game for children? (must collect)
6.0 cancellation of member registration verification code
JS foundation -- hijacking of this keyword
Interview question -- event cycle
2022-07-19: all factors of F (I): I are added up after each factor is squared. For example, f (10) = 1 square + 2 square + 5 square + 10 square = 1 + 4 + 25 + 100 = 130.
Reasons for not sending requests after uni app packaging
Riotboard development board series notes (VII) -- the use of framebuffer
Error: tomee required to support ear/ejb deployment