当前位置:网站首页>[Presto] Presto parameter configuration optimization
[Presto] Presto parameter configuration optimization
2022-07-06 11:31:00 【kiraraLou】
Preface
Former company presto The service has not been very stable , With resource group , Configuration optimization , Code secondary development .presto The service has basically stabilized . This record presto What optimization has been done to service parameter configuration .
The production environment here JVM
Are configured as 40G.
config.properties
- Disable reservation pool
Reserved Pool : When you have one worker Of General Pool Out of memory ,reserved pool Will work . This is the time coordinator The cluster will select the query that consumes the most memory , And assign the query to all worker Of reserved pool.
Reserved Pool The value is equal to the query.max-total-memory-per-node Size . query.max-total-memory-per-node Is a single node User memory and system memory Maximum occupancy .
and Reserved Pool It is reserved by the system at startup , Moving together will occupy , So now there's a question , In reality , We seldom use Reserved Pool, And it also takes up a lot of memory resources .
So we can configure parameters
experimental.reserved-pool-enabled=false
- query.max-memory-per-node
In a single worker The largest one that can be used above user memory value ( Default JVM max memory * 0.1) To adjust to JVM max memory * 0.25
query.max-memory-per-node=10GB
- query.max-total-memory-per-node
Single Query In a single Worker The maximum allowed on the user memory + system memory To adjust to JVM max memory * 0.4
query.max-total-memory-per-node=16GB
- memory.heap-headroom-per-node
This memory is mainly the memory allocation of third-party libraries , Unable to track statistics . ( Default JVM max memory * 0.3) To adjust to JVM max memory * 0.2
memory.heap-headroom-per-node=8GB
- query.max-memory
The maximum memory that a single query can use instantly on all task scheduling nodes ( Peak memory for a single query ) To adjust to <= query.max-total-memory-per-node * workers * 0.8
query.max-memory=192GB
- query.low-memory-killer.policy
When presto Occurrence cluster OOM Memory protection strategy when , To configure total-reservation yes kill Lose the task that occupies the most memory .
query.low-memory-killer.policy=total-reservation
- More configuration
# Extend the waiting time
exchange.http-client.request-timeout=10s
# From the other Presto The number of threads that the node obtains data . Higher values can improve the performance of large clusters or clusters with high concurrency ( The default value is :25)
exchange.client-threads=50
exchange.http-client.idle-timeout = 10s
# Parallel operator ( For example, join and aggregate ) Default local concurrency . Lower values are better for clusters that run many queries at the same time ( The default value is :16, Must be 2 The power of )
task.concurrency=2
task.max-worker-threads=60
# You can create for processing HTTP The maximum number of threads responding . On a cluster with a large number of concurrent queries or on a cluster with hundreds or thousands of workers , It can be raised ( The default value is :100)
task.http-response-threads=200
join-distribution-type=AUTOMATIC
node-scheduler.max-splits-per-node=200
query.max-stage-count=400
# Automatically kill Run longer than 20 Minutes of sql( Replace script kill Mission , Give Way presto Automatic management ):
query.max-run-time=1200s
# Solve the new version remote too large Report errors
exchange.http-client.max-content-length=128MB
node-manager.http-client.max-content-length=64MB
jvm.config
-Xmx40G
-Xms40G
-XX:-UseBiasedLocking
-XX:+UseG1GC
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseGCOverheadLimit
-XX:OnOutOfMemoryError=kill -9 %p
-DHADOOP_USER_NAME=hive
-Duser.timezone=Asia/Shanghai
-Djdk.attach.allowAttachSelf=true
-Djava.security.krb5.conf=/etc/krb5.conf
-XX:G1ReservePercent=15
-XX:InitiatingHeapOccupancyPercent=40
-XX:ConcGCThreads=8
边栏推荐
- Valentine's Day flirting with girls to force a small way, one can learn
- Nanny level problem setting tutorial
- 2019腾讯暑期实习生正式笔试
- Error reporting solution - io UnsupportedOperation: can‘t do nonzero end-relative seeks
- Deoldify项目问题——OMP:Error#15:Initializing libiomp5md.dll,but found libiomp5md.dll already initialized.
- 【kerberos】深入理解kerberos票据生命周期
- QT creator runs the Valgrind tool on external applications
- Attention apply personal understanding to images
- ImportError: libmysqlclient. so. 20: Cannot open shared object file: no such file or directory solution
- SQL时间注入
猜你喜欢
[number theory] divisor
Vs2019 first MFC Application
Case analysis of data inconsistency caused by Pt OSC table change
Nanny level problem setting tutorial
About string immutability
Machine learning -- census data analysis
解决安装Failed building wheel for pillow
QT creator design user interface
Dotnet replaces asp Net core's underlying communication is the IPC Library of named pipes
Integration test practice (1) theoretical basis
随机推荐
Test objects involved in safety test
2020网鼎杯_朱雀组_Web_nmap
AcWing 242. A simple integer problem (tree array + difference)
Cookie setting three-day secret free login (run tutorial)
【kerberos】深入理解kerberos票据生命周期
自动机器学习框架介绍与使用(flaml、h2o)
AcWing 179. Factorial decomposition problem solution
數據庫高級學習筆記--SQL語句
vs2019 第一个MFC应用程序
Double to int precision loss
{one week summary} take you into the ocean of JS knowledge
Punctual atom stm32f103zet6 download serial port pin
2019腾讯暑期实习生正式笔试
Vs2019 first MFC Application
Aborted connection 1055898 to db:
安装numpy问题总结
【CDH】CDH/CDP 环境修改 cloudera manager默认端口7180
QT creator custom build process
{一周总结}带你走进js知识的海洋
L2-001 emergency rescue (25 points)