Flink SQL task taskmanager memory settings

Submitting flink When the task , adopt taskmanager.memory.process.size To configure taskmanager Total running memory of , It is sometimes unreasonable to divide the detail memory according to the default proportion . As shown below ：

Configured with 10G Of taskmanager Total running memory , By default Managed Memory Allocated 3.5G Of memory , The actual task does not need so much memory to run this part . How to maximize memory management and configuration is very important .
This paper mainly introduces FlinkSQL The task of TaskManager Memory tuning , other flink The task can also be used as a reference .

1、 start-up flink sql client

sql-client.sh embedded shell

You can start with flink The client executes the following commands flinksql Task allocation .

# Set up flink Task itself task name SET pipeline.name = 'flink-sql-test';# Set up flink on yarn Task name for SET yarn.application.name= 'flink-sql-test';# Set up flink checkpoint File storage address SET state.checkpoints.dir = 'hdfs:///flink/checkpoints/flink-sql-test';# Set up taskmanager The total running memory of is 5GSET taskmanager.memory.process.size = '5120m';# Set up flink Frame memory SET taskmanager.memory.framework.heap.size = '128m';# Set user code running memory SET taskmanager.memory.task.heap.size = '4096m';# Set up managed memory Memory SET taskmanager.memory.managed.size = '128m';# Set the off heap memory SET taskmanager.memory.framework.off-heap.size = '128m';# Set up network cache SET taskmanager.memory.network.max = '128m';# Set up JVM Memory SET taskmanager.memory.jvm-metaspace.size = '256m';SET taskmanager.memory.jvm-overhead.max = '256m';# Set up jobmanager Total memory SET jobmanager.memory.process.size= '1024m';# Set up flink on yarn Of yarn queue SET yarn.application.queue= 'other';

Submitted after debugging flink taskmanager The runtime memory is as follows ：

You can see the original configuration 10G taskmanager Memory , Now it's just a matter of 5G You can run , Memory optimized by half , The cluster resources are effectively utilized .
of flink Memory tuning configuration , Students who need it can also refer to this article for more detailed memory configuration flink Task memory tuning ,TaskManager、JobManager Memory configuration _ Big data fishing blog -CSDN Blog
author ： Big data fishing

Game programming , A game development favorite ~

If the picture is not displayed for a long time , Please use Chrome Kernel browser .