Flink learning 1: Introduction

flink Catalog ：

 

1. Traditional data processing mode ：

1.1 Central database schema

The central database is heavily loaded , And once the central database has problems , All business systems will crash

1.2 lamda Data warehouse mode

however lamda Data warehouse of mode , Generally, relational database is used , Can not meet the storage of massive data

 1.3 be based on Hadoop Of hdfs Built lamda Data warehouse mode

To some extent, it solved , Different calculation modes （ Real time and offline ） Business needs .

however , This flow batch mode , It is equivalent to real-time and offline architectures , Lead to High complexity and operation and maintenance cost .

 1.4 Stream processing based pattern

Stream processing architectures are generally divided into 2 part , Message transmission layer （ Responsible for collecting new data and Push data ） and Stream processing layer （ Responsible for data conversion and processing ）

The general framework of stream processing

flink There is no central database , And stream processing naturally supports batch processing （ No longer need two real-time and offline architectures ）

2.flink Introduction to

2.1 flink Characteristics

 

 

 flink The advantages of ：

2.2 flink The advantages of ：

2.2.1 Streaming window ：

Among them the first 3 A highly flexible streaming window

Because there is no end in streaming data , You can't calculate directly , therefore flink Put forward , The concept of windows ：

  The concept of windows

The classification of windows ：

 2.2.2 The state of streaming computing ：

  Stateful means that the state of each event is recorded ：

2.2.3 Good fault tolerance

How to ensure distributed systems , Consistency of all nodes

 flink Keep creating snapshots , To compare data consistency

2.2.4 High performance memory management

because java There will be some problems in memory management , therefore flink Created your own memory management

 

 flink How to manage memory by yourself

Sequence and deserialize all objects , Store... In memory

Serialize storage memory , The benefits of doing this ：

 2.2.5 Support iteration and incremental iteration

Iteration here refers to ： The result of each iteration , To the next iteration

Incremental iteration ： The next iteration only needs , Calculate part of the data of the last iteration , perhaps Only part of the data set from the last iteration needs to be updated