The source code of the daily book analyzes the design idea of Flink and solves the problems in Flink

From the emergence to popularity of stream computing , Experienced a lot of changes —— From the early days Apache Storm And other technologies , Up to now, more and more companies choose to use Apache Flink As the core technology of stream processing .Flink With its powerful batch flow integrated processing capability and low latency 、 High throughput , It is attracting more and more companies and users to join Flink Community .

 

For developers who want to learn and master a technology , most important of all Start with the principle , Step by step 、 Learn from simple to profound , then Then slowly go deep into the internal implementation of Technology .

Introduce a book that will Flink A book with clear source code 《Flink Design and implementation ： Core principles and source code analysis 》, Help those who want to understand the source code 、 Deep Mastery Flink Readers who realize the underlying core technology but don't have much time to study . This book can help readers more easily Flink This technology is applied to practical work .

Learning the source code can not only improve the technical foundation , It can also deepen the understanding of Technology . To master a technology in depth , It can be said that there is no more effective way than reading the source code . Usually , Reading the source code has a high technical threshold , Not easy to start , We need to have a certain understanding and understanding of Technology , At least be able to apply it in practical work very skillfully , In order to better understand its underlying operating principle . Otherwise, without knowing the use of the framework, you can learn the source code implementation rashly , It is very easy to fall into confusion and confusion , This greatly affects the learning experience .

This book will introduce from many aspects Flink Principle implementation and source code , Include Flink The design and implementation of various programming interfaces and the internal principles of cluster runtime .

The whole book consists of 8 Chapter ：

• The first 1 Chapter Introduce Flink Design concept and basic structure ;

• The first 2 Chapter Introduce DataStream Design and implementation ;

• The first 3 Chapter Introduce the core principle and implementation of runtime , Include Dispatcher、ResourceManager as well as JobManager And other core components of the source code level analysis and introduction ;

• The first 4 Chapter Introduce Flink The overall process of task submission and execution , Including client implementation 、 Runtime job execution process 、JobGraph And ExecutionGraph Graph conversion, etc ;

• The first 5 Chapter Introduce different cluster deployment modes , Include On Yarn、On Kubernetes etc. ;

• The first 6 Chapter Introduce state management and fault tolerance , Including the design and implementation of different types of state backend ;

• The first 7 Chapter Introduce Flink Network communication , Include RPC Communication and based on Netty Implemented network stack ;

• The first 8 Chapter Introduce Flink memory management , Include MemorySegment Design and implementation of .

Statement ： In this paper, from “ Huazhang computer ” official account