Flink Learning 11: Flink Program Parallelism

1. Introduction to program parallelism

A slot is a resource group,

Generally, by default, the number of slots on each node is the number of CPU cores on the node.

flink will split a task into multiple subtasks,
and then the subtasks will be executed by multiple parallel threads.

The number of parallel threads of a task is called the parallelism of the task.

flink will assign subtasks to slot execution, so the maximum parallelism of the task is determined by the number of slots available on each TaskManager.

For example, there are 4 slots on a taskManager, then the taskManager will allocate 25% of the memory for each slot.

One or more threads can run in the same slot.

Threads in the same slot share the same Java virtual machine JVM

Tasks in the same JVM share TCP connection and heartbeat information

A slot of Task Manager represents an available thread, ps: slot only corresponds to memory isolation, not CPU isolation

By default, flink allows subtasks to share slots, even if they are subtasks of different tasks, but if they are from the same job.This sharing can achieve better resource utilization.

2. Introduction to parallelism at the execution level

After generating the environment as above, set setParallelism to 1, which means to set the parallelism to 1.