当前位置:网站首页>【presto】presto 参数配置优化
【presto】presto 参数配置优化
2022-07-06 09:15:00 【kiraraLou】
前言
之前公司的 presto 服务一直不是很稳定,随着资源组,配置优化,代码二次开发。presto 服务基本上已经趋于稳定了。这个记录下 presto 服务参数配置都做过哪些优化。
这边生产环境的 JVM
都是配置为 40G。
config.properties
- 禁用保留池
Reserved Pool : 当有一个worker的General Pool 内存耗尽,reserved pool将会发挥作用。这个时候coordinator 将会选出集群出消耗内存最大的查询,并将该查询分配给所有worker的reserved pool。
Reserved Pool的值等于query.max-total-memory-per-node的大小。 query.max-total-memory-per-node是单个节点User memory 和system memory占用的最大值。
而且Reserved Pool是启动时系统预留的,一起动就会占用,所以现在有一个问题,在实际中的情况来讲,我们会极少使用到Reserved Pool,而且它还占用了很多的内存资源。
所以我们可以通过配置参数
experimental.reserved-pool-enabled=false
- query.max-memory-per-node
在单个 worker 上面可以使用的最大的user memory值 (默认JVM max memory * 0.1)调整到 JVM max memory * 0.25
query.max-memory-per-node=10GB
- query.max-total-memory-per-node
单个Query在单个Worker上允许的最大user memory + system memory 调整到 JVM max memory * 0.4
query.max-total-memory-per-node=16GB
- memory.heap-headroom-per-node
这个内存主要是第三方库的内存分配,无法统计跟踪。 (默认JVM max memory * 0.3)调整到 JVM max memory * 0.2
memory.heap-headroom-per-node=8GB
- query.max-memory
单个查询在所有任务调度的节点上瞬间最大能用的内存(单个查询的峰值内存) 调整到 <= query.max-total-memory-per-node * workers * 0.8
query.max-memory=192GB
- query.low-memory-killer.policy
当presto集群发生OOM时的内存保护策略,配置 total-reservation 是kill 掉占用内存最大的任务。
query.low-memory-killer.policy=total-reservation
- 更多配置
#延长等待时间
exchange.http-client.request-timeout=10s
#从其他Presto节点获取数据的线程数。较高的值可以提高大型群集或具有很高并发性的群集的性能(默认值:25)
exchange.client-threads=50
exchange.http-client.idle-timeout = 10s
#并行运算符(例如联接和聚合)的默认本地并发性。较低的值对于同时运行许多查询的集群更好(默认值:16,必须是2的幂次)
task.concurrency=2
task.max-worker-threads=60
#可以创建用于处理HTTP响应的最大线程数。在具有大量并发查询的群集上或在具有数百或数千个工作程序的群集上,可以调高(默认值:100)
task.http-response-threads=200
join-distribution-type=AUTOMATIC
node-scheduler.max-splits-per-node=200
query.max-stage-count=400
#自动kill运行时长超过20分钟的sql(代替脚本kill任务,让presto 自动管理):
query.max-run-time=1200s
#解决新版本 remote too large 报错
exchange.http-client.max-content-length=128MB
node-manager.http-client.max-content-length=64MB
jvm.config
-Xmx40G
-Xms40G
-XX:-UseBiasedLocking
-XX:+UseG1GC
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseGCOverheadLimit
-XX:OnOutOfMemoryError=kill -9 %p
-DHADOOP_USER_NAME=hive
-Duser.timezone=Asia/Shanghai
-Djdk.attach.allowAttachSelf=true
-Djava.security.krb5.conf=/etc/krb5.conf
-XX:G1ReservePercent=15
-XX:InitiatingHeapOccupancyPercent=40
-XX:ConcGCThreads=8
边栏推荐
- Are you monitored by the company for sending resumes and logging in to job search websites? Deeply convinced that the product of "behavior awareness system ba" has not been retrieved on the official w
- PHP - whether the setting error displays -php xxx When PHP executes, there is no code exception prompt
- ImportError: libmysqlclient. so. 20: Cannot open shared object file: no such file or directory solution
- Ansible practical series I_ introduction
- Software testing - interview question sharing
- 安装numpy问题总结
- 02 staff information management after the actual project
- C语言读取BMP文件
- ES6 Promise 对象
- Django running error: error loading mysqldb module solution
猜你喜欢
引入了junit为什么还是用不了@Test注解
How to build a new project for keil5mdk (with super detailed drawings)
Case analysis of data inconsistency caused by Pt OSC table change
Summary of numpy installation problems
QT creator create button
Machine learning notes week02 convolutional neural network
Dotnet replaces asp Net core's underlying communication is the IPC Library of named pipes
wangeditor富文本引用、表格使用问题
Valentine's Day flirting with girls to force a small way, one can learn
Machine learning -- census data analysis
随机推荐
Heating data in data lake?
[BSidesCF_2020]Had_a_bad_day
Attention apply personal understanding to images
机器学习--人口普查数据分析
L2-007 family real estate (25 points)
Codeforces Round #771 (Div. 2)
How to build a new project for keil5mdk (with super detailed drawings)
Introduction and use of automatic machine learning framework (flaml, H2O)
Punctual atom stm32f103zet6 download serial port pin
AcWing 1298.曹冲养猪 题解
AcWing 179.阶乘分解 题解
{一周总结}带你走进js知识的海洋
Learn winpwn (3) -- sEH from scratch
Error connecting to MySQL database: 2059 - authentication plugin 'caching_ sha2_ The solution of 'password'
neo4j安装教程
ImportError: libmysqlclient. so. 20: Cannot open shared object file: no such file or directory solution
ES6 promise object
Error reporting solution - io UnsupportedOperation: can‘t do nonzero end-relative seeks
Unable to call numpy in pycharm, with an error modulenotfounderror: no module named 'numpy‘
C语言读取BMP文件