当前位置:网站首页>Spark学习:为Spark Sql添加自定义优化规则
Spark学习:为Spark Sql添加自定义优化规则
2022-07-31 13:03:00 【我爱夜来香A】
一、自定义优化规则
Spark在2.2版本引入了一个强大的特性,添加钩子和拓展点,允许用户自定义优化规则
1、实现自定义规则 (静默规则,通过 set spark.sql.planChangeLog.level=WARN,确认执行到就行)
case class MyPushDown(spark: SparkSession) extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
.... }
}
2、创建自己的 Extension 并注入
class MySparkSessionExtension extends (SparkSessionExtensions => Unit) {
override def apply(extensions: SparkSessionExtensions): Unit = {
extensions.injectOptimizerRule {
session =>
new MyPushDown(session)
}
}
}
3、通过 spark.sql.extensions 提交
bin/spark-sql --jars my.jar --conf MySparkSessionExtension
二、代码实现
1、pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Spark</groupId>
<artifactId>Spark</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>compile-scala</id>
<phase>compile</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>test-compile-scala</id>
<phase>test-compile</phase>
<goals>
<goal>add-source</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>2.12.15</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass></mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>java</executable>
<includeProjectDependencies>true</includeProjectDependencies>
<includePluginDependencies>false</includePluginDependencies>
<classpathScope>compile</classpathScope>
<mainClass>cn.spark.study.App</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
2、class
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.rules.Rule
case class MyRule(spark: SparkSession) extends Rule[LogicalPlan] {
logWarning("成吉思汗的规则")
override def apply(plan: LogicalPlan): LogicalPlan = plan
}
import org.apache.spark.sql.SparkSessionExtensions
class MySparkSessionExtension extends (SparkSessionExtensions => Unit) {
override def apply(extensions: SparkSessionExtensions): Unit = {
extensions.injectOptimizerRule {
session => MyRule(session)
}
}
}
3、使用maven打成jar包
三、结果验证
大功告成!完美!!!
边栏推荐
- 亲测可用!!!WPF中遍历整个窗口的所有TextBox组件,对每个输入框做非空判断。
- 阿里三面:MQ 消息丢失、重复、积压问题,怎么解决?
- Banyan Tree Loan GPU Hardware Architecture
- Ali on three sides: MQ message loss, repetition, backlog problem, how to solve?
- 滑窗法切分数据
- 系统集成项目管理工程师(软考中级)知识点总结【挣值分析】【关键路径】
- 【CPU设计实战】简单流水线CPU设计
- SAP 电商云 Spartacus SSR Optimization Engine 几处 timeout 的执行顺序
- The function of SQL GROUP BY dependence
- Hard disk partition, expand disk C, no reshipment system, not heavy D dish of software full tutorial.
猜你喜欢
PyQt5 rapid development and actual combat 10.1 Get city weather forecast
CentOS7 - yum install mysql
IDEA版Postman插件Restful Fast Request,细节到位,功能好用
log4j2的使用
Istio微服务治理网格的全方面可视化监控(微服务架构展示、资源监控、流量监控、链路监控)
networkx绘制度分布
PyQt5快速开发与实战10.2 复利计算 && 10.3 刷新博客点击量
网络协议及相关技术详解
ECCV2022: Recursion on Transformer without adding parameters and less computation!
EXCEL如何快速拆分合并单元格数据
随机推荐
使用openssl命令生成证书和对应的私钥,私钥签名,公钥验签
基于神经网络的多柔性梁耦合结构振动控制
分布式监视 Zabbix 和 Prometheus 到底怎么选?千万别用错了!
Hard disk partition, expand disk C, no reshipment system, not heavy D dish of software full tutorial.
Exploring Plain Vision Transformer Backbones for Object Detection Paper Reading Notes
AMBA APB学习记录(AMBA 2.0)
alert(1) (haozi.me)靶场练习
知名无人驾驶公司:文远知行内推
WPF中报错:“未将对象引用设置到对象的实例。”
【牛客刷题-SQL大厂面试真题】NO3.电商场景(某东商城)
JSP中如何借助response对象实现页面跳转呢?
C#使用NumericUpDown控件
centos7安装mysql5.7
IDEA如何运行web程序
C#控件 ToolStripProgressBar 用法
Verilog——基于FPGA的贪吃蛇游戏(VGA显示)
六石编程学:不论是哪个功能,你觉得再没用,会用的人都离不了,所以至少要做到99%
Encapsulation of conversion between Json and objects (Gson)
The cluster of safe mode
查看Oracle数据库的用户名和密码