当前位置:网站首页>Spark学习:用spark实现ETL
Spark学习:用spark实现ETL
2022-07-30 18:45:00 【我爱夜来香A】
一、RDBMS To RDBMS
1、pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Spark</groupId>
<artifactId>Spark</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>2.12.15</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass></mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>cn.spark.study.App</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
2、Main
package bigData
import org.apache.spark.sql.SparkSession
import java.util.Properties
object DataSync {
def main(args:Array[String]):Unit = {
val app = s"${
this.getClass.getSimpleName}".filter(!_.equals('$'))
val spark: SparkSession = SparkSession.builder
.appName(app)
.master("local[*]")
.config("spark.shuffle.consolidateFiles", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.streaming.kafka.maxRatePerPartition", "500")
.config("spark.streaming.stopGracefullyOnShutdown", "true")
.config("spark.network.timeout", "600")
.config("spark.streaming.kafka.consumer.poll.ms", "60000")
.config("spark.core.connection.ack.wait.timeout", "900")
.config("spark.rpc.message.maxSize", "50")
.config("spark.akka.timeout", "900")
.getOrCreate()
//数据源
val source_url = s"jdbc:oracle:thin:@192.168.0.101:1521/orcl"
val source_prop = new Properties()
source_prop.put("user","scott")
source_prop.put("password","scott")
source_prop.put("driver","oracle.jdbc.driver.OracleDriver")
//目标库
val target_url = s"jdbc:oracle:thin:@192.168.0.101:1521/orcl"
val target_prop = new Properties()
target_prop.put("user","scott")
target_prop.put("password","scott")
target_prop.put("driver","oracle.jdbc.driver.OracleDriver")
import spark.implicits._
spark.read.jdbc(source_url,"EMP",source_prop).write.jdbc(target_url,"EMP_TEST",target_prop)
}
}
3、运行结果
不过要注意,代码中没有做表已经存在的判断,第二次运行会报错
边栏推荐
- "Ruffian Heng Embedded Bimonthly" Issue 59
- 【Pointing to Offer】Pointing to Offer 22. The kth node from the bottom in the linked list
- arcpy获取要素类(属性表)包含的数目
- Mongo for infrastructure
- node封装一个控制台进度条插件
- Codeblocks + Widgets 创建窗口代码分析
- 【剑指 Offe】剑指 Offer 17. 打印从1到最大的n位数
- Two-point answer naked question (plus a little pigeonhole principle)
- ESP8266-Arduino编程实例-HC-SR04超声波传感器驱动
- AWS 控制台
猜你喜欢
网络基础(二)-Web服务器-简介——WampServer集成服务器软件之Apache+MySQL软件安装流程 & netstat -an之检测计算机的端口是否占用
运营 23 年,昔日“国内第一大电商网站”黄了...
Does the satellite phone communicate directly with the satellite or through a ground station?
【剑指 Offer】剑指 Offer 22. 链表中倒数第k个节点
6 yuan per catty, why do Japanese companies come to China to collect cigarette butts?
C# wpf borderless window add shadow effect
一文读懂“语言模型”
微博广告分布式配置中心的构建与实践(有彩蛋)
SwiftUI iOS 精品开源项目之 完整烘焙食品菜谱App基于SQLite(教程含源码)
cocos creater 热更重启导致崩溃
随机推荐
Deepen school-enterprise cooperation and build an "overpass" for the growth of technical and skilled talents
Chapter 4 Controlling the Execution Flow
软件测试13年从业经验的前辈,总结的5条测试就业建议....
Mysql执行原理剖析
LocalDate时间生成
高精度加法
CCNA-ACL(访问控制列表)标准ACL 扩展ACL 命名ACL
Anaconda Navigator卡在loading applications
kotlin by lazy
C# wpf borderless window add shadow effect
CCNA-网络汇总 超网(CIDR) 路由最长掩码匹配
Critical Reviews | 南农邹建文组综述全球农田土壤抗生素与耐药基因分布
[OC study notes] attribute keyword
3D机器视觉厂商的场景争夺战役
[Use of Qt Designer tool]
卫星电话是直接与卫星通信还是通过地面站?
中集世联达飞瞳全球工业人工智能AI领军者,全球顶尖AI核心技术高泛化性高鲁棒性稀疏样本持续学习,工业级高性能成熟AI产品规模应用
怎么样的框架对于开发者是友好的?
Meta元宇宙部门第二季度亏损28亿!仍要继续押注?元宇宙发展尚未看到出路!
荐号 | 对你有恩的人,不要请吃饭来报答