Simple record of Flink principle flow chart

Record some pictures for future review

Yarn-per-job startup flowchart

PRC Flowchart

Akka Actor-based RPC communication system, the core concept is alsoMessage, it is based on coroutines and has good performance: scala-based partial functions, high ease of use,
But it is only RPC communication after all, and cannot be used for data transmission of large packages/streams, so more usage scenarios are stillNetty

Task Scheduling

streamGraph: Concatenates operators according to the order in the client, encapsulating streams and edges.Operators that do not convert data become edges, such as keyBy becomes hash StreamEdgeJobGraph: Optimize multiple operation chains.Multiple StreamNodes may be chained into a JobVertex after optimization.and intermediate datasets with additional vertices

ExecutionGraph: The execution vertices are expanded according to the degree of parallelism, and the intermediate result set becomes multiple intermediate result partitions

The scheduler performs RPC through the gateway to open and schedule real physical execution

Memory Management

Memory Model

p>

JVM metaspace: default 256mbJVM execution overhead: default total memory 0.1 ratioFramework memory: The memory occupied by TaskManager itself is not included in slot resources.The default heap is 128mb inside and outside the heapTask memory: Off-heap memory is not enabled by default, and on-heap memory defaults to none(It is obtained by deducting other memory from Flink memory, that is, all the extra memory belongs to Task memory, Flink memory = total memory - JVM metaspace and execution overhead)Network memory: External memory for network data exchange.The default size is Flink memory ratio 0.1/minimum 64mb/maximum 1GManaged memory: The default Flink memory ratio is 0.4

Memory Data Structures - Memory Segments

Memory management for network transfers

Transfer data: The data enters the local buffer pool, then enters the network buffer pool, and transmits data through the network buffer poolReceive data: the process is reversed