Flink's fault tolerance mechanism (checkpoint)

Flink Reliability cornerstone of -Checkpoint Fault tolerance mechanism

1. summary

flink Of checkpoint The mechanism can guarantee Flink When an operator fails for some reason in the whole cluster , It can restore the state of the whole application flow graph to a certain state before the failure , Ensure the state consistency of the application flow graph .Flink Of Checkpoint Mechanism principle from “Chandy-Lamport algorithm” Algorithm .

2. principle

Every need Checkpoint At startup ,Flink Of JobManager Create a CheckpointCoordinator（ Checkpoint Coordinator ）,CheckpointCoordination Take full responsibility for the snapshot of this application .

The process ：

（1）CheckpointCoordinator( Checkpoint Coordinator ) Periodically apply all source Operator sending barrier( barrier ).

（2） When a source Operator receives a barrier when , Data processing is suspended , Then make a snapshot of your current state , And save to the specified persistent storage , Finally to CheckpointCoordinator Report on your snapshot production , At the same time broadcast to all its downstream operators barrier, Recovery data processing

（3） Downstream operator received barrier after , Will pause its own data processing , Then make a snapshot of the relevant state , And save to the specified persistent storage , Finally to CheckpointCoordinator Report on your own snapshot , At the same time broadcast to all its downstream operators barrier, Recovery data processing .

（4） Each operator follows the steps 3 Continuous snapshot and broadcast to downstream , Until the last barrier Pass on to sink operator , Snapshot completed .

（5） When CheckpointCoordinator After receiving the report of all operators , Consider that the snapshot of this cycle is made successfully ; otherwise , If you don't receive all operator reports within the specified time , The snapshot of this cycle is considered to be failed .

3.Flink Of Checkpoint And Spark Compared with ,Flink Is there any difference or advantage ？

Spark Streaming Of Checkpoint Only for Driver The recovery of data and metadata is done Checkpoint. and Flink Of Checkpoint The mechanism is much more complicated , It uses lightweight distributed snapshots , The snapshot of each operator is implemented , And a snapshot of the data in the flow .