Summarize Flink runtime architecture in simple terms

One 、Flink Introduction to various components of runtime

Flink  The runtime architecture consists of four different components , They work together as you run the stream processing application ：

Job manager （JobManager）

、

Explorer （ResourceManager）

、

Task manager （TaskManager）

, as well as

The dispenser （Dispatcher）

. because  Flink  Yes, it is  Java  and  Scala  Realized , So all the components will run in Java  Virtual machine . Next, the functions of each component are briefly introduced i.

Job manager （JobManager）

The job manager will control one of its main processes , Each application will have a different JobManager  Controlled execution .JobManager  To the explorer （ResourceManager） Request the resources necessary to perform the task , The task manager （TaskManager） Slot on （slot）. Once it gets enough resources , The execution diagrams are distributed to the... That actually run them TaskManager  On . And in the process of running ,JobManager  Will be responsible for all operations that need central coordination , Check points, for example （checkpoints） The coordination of .

Explorer （ResourceManager）

Mainly responsible for managing the task manager （TaskManager） Slot for （slot）,TaskManger  The slot is  Flink  The processing resource unit defined in .Flink  Different resource managers are provided for different environments and resource management tools , such as YARN、Mesos、K8s, as well as  standalone  Deploy . When  JobManager  When applying for slot resources ,ResourceManager There will be free slots  TaskManager  Assigned to  JobManager. If  ResourceManager  There are not enough slots for  JobManager  Request , It can also initiate a session to the resource provider platform , To provide start-up  TaskManager Process container . in addition ,ResourceManager  Also responsible for terminating idle  TaskManager, Free computing resources .

Task manager （TaskManager）

Flink  The process of work in . Usually in  Flink  There will be more than one  TaskManager  function , every last  TaskManager Both contain a certain number of slots （slots）. The number of slots limits  TaskManager  Number of tasks that can be performed . After starting ,TaskManager  Will register its slot with the resource manager ; After receiving the instruction from the resource manager ,TaskManager  One or more slots will be provided to  JobManager  call .JobManager  You can assign tasks to slots （tasks） Here we go . In the process of execution , One  TaskManager  Can run the same application with other  TaskManager  Exchange data .

The dispenser （Dispatcher）

Can run across jobs , It provides... For application submission  REST  Interface . When an application is submitted for execution , The distributor will start and hand over the application to a  JobManager. Because it is  REST  Interface , therefore  Dispatcher  It can be used as a cluster  HTTP  Access point , In this way, it can not be blocked by the firewall .Dispatcher  It will also activate a  Web UI, Information used to easily display and monitor job execution .Dispatcher  It may not be necessary in the architecture , It depends on how the application is submitted to run .

Two 、Flink The process of task submission

That's all flink Functions corresponding to each component of the runtime architecture , Next, let's take a look at how they work together with each other .

General interaction and cooperation diagram ：

To submit a flink Program to distributor , The dispenser （Dispatcher） Hand over the submitted application to the task manager （JobManager）, Then to the resource manager （ResourceManager） Request the resources necessary to perform the task , Then the resource manager will have free slots  TaskManager  Assigned to  JobManager, then JobManager The task to be performed in the slot will be submitted to TaskManager .

Specific to the Yarn Interaction diagram on

• Client  Submit for  Job  The client of , It could be running on any machine （ And  JobManager  The environment is connected ）. Submit  Job  after ,Client  You can end the process （Streaming  The task of ）, You can also not end and wait for the results to return .

• JobManager  Lord   want   negative   responsibility   transfer   degree  Job  and   Association   transfer  Task  do  checkpoint,  job   responsibility   On   very   image Storm  Of  Nimbus. from  Client  I received it from you  Job  and  JAR  After the package and other resources , An optimized execution plan will be generated , And  Task  The unit is scheduled to each  TaskManager  To carry out .

• TaskManager  The slot number is set at the start-up time （Slot）, Every  slot  Can start a Task,Task  For threads . from  JobManager  Receive what needs to be deployed  Task, After deployment starts , Build with your own upstream  Netty  Connect , To receive data and process

3、 ... and 、Flink Principle of task scheduling

Flink  Set   Group   Qi   dynamic   after  ,  The first   First   Meeting   Qi   dynamic   One   individual  JobManger  And one or more of TaskManager. from  Client  Submit a task to  JobManager,JobManager  Then schedule tasks to each TaskManager  To carry out , then  TaskManager  Report heartbeat and statistics to  JobManager.TaskManager  Data transmission in the form of stream . All three are independent  JVM  process .

TaskManger  And  Slots

• One  Worker（TaskManager） It's just one.  JVM  process , Have one or more... Inside  Task Slot  Further subdivide the process  CPU  resources .

• 

• Slot  Refer to  TaskManager  Maximum ability to execute concurrently ->taskmanager.numberOfTaskSlots->ys

• parallelism  Refer to  TaskManager  The concurrency capability actually used ->parallelism.default->p

• same Slot  Threads in share the same  JVM.  same  JVM  Task sharing in  TCP  Connection and heartbeat messages .TaskManager  One of the  Slot  Represents an available thread , This thread has fixed memory , Be careful  Slot  Only memory isolation , No, right  CPU  Isolation

Let's say there are  3  individual  TaskManager, every last  TaskManager  Distribution in  3  individual TaskSlot, That is, every one of them  TaskManager  You can receive  3  individual  task, altogether  9  individual  TaskSlot, If we set  parallelism.default=1, That is, the default parallelism of the running program is  1,9  individual  TaskSlot  It only took  1 individual , Yes  8  A free time , therefore , Setting the right parallelism can improve efficiency

Programs and data streams （DataFlow）

• be-all  Flink  The program is composed of three parts ： Source 、Transformation  and  Sink.Source  Responsible for reading data source ,Transformation  Processing with various operators ,Sink  Responsible for output .

• Flink  The program that runs on will be mapped to “ Logical data flow ”（dataflows）, It contains these three parts . every last  dataflow  With one or more  sources  Start with one or more  sinks  end .dataflow  Similar to any directed acyclic graph （DAG）.

• The conversion operation in the program （transformations） Follow  dataflow  Medium （operator） It's a one-to-one relationship , But sometimes , One  transformation  May correspond to multiple  operator.

Parallelism （Parallelism）

Flink  The execution of the program has parallel 、 Distributed features . Subtasks of a particular operator （subtask） The number of is called its parallelism （parallelism）. Parallelism of a stream program , It can be considered as the largest parallelism of all its operators . In a program , Different operators may have different degrees of parallelism .

Stream  There are two modes of transferring data between operators .

• One-to-one：stream( For example  source  and  map operator  Between ) Maintaining the order of partitions and elements . That means  map  The number and order of elements seen by the subtask of the operator follow  source  The number of elements produced by the subtask of the operator 、 Same order ,map、fliter、flatMap  And so on  one-to-one  Correspondence of .

• Redistributing：stream(map() Follow  keyBy/window  Between or  keyBy/window  Follow  sink Between ) The partition of will change . The subtasks of each operator depend on the selected  transformation  Send data to different target tasks . for example ,keyBy()  be based on  hashCode  Repartition 、broadcast  and  rebalance It's going to be randomly repartitioned , These operators all cause  redistribute  The process

Task chain （Operator Chains）

Of the same degree of parallelism  one to one  operation ,Flink  In this way, the connected operators are linked together to form a  task, The original operator becomes part of it . Link the operator to  task  It's a very effective optimization ： It can reduce the switching between threads and cache based data exchange , Improve throughput while reducing latency .