Flink learning 2: application scenarios

Catalog ：

 

1 Three application scenarios

1.1 Event driven

Business Driven ： Computing and storage are in the same place .

event Driven ： Computing and storage are separate . Applications with state , Read events from one or more event streams , And react （ Trigger calculation , Status update, etc ）

Transaction driven model

The event driver is , Read and write data from the remote transaction database . Event driven applications are evolved from traditional applications .

Typical event driven application scenarios ：

Advantages of event driven applications ：

Faster access to local data

  Event driven applications , Because it reads local data , So just consider your own data , It will not involve the modification of other systems .

1.2 flink How to support event driven applications ：

An important capability of event handling is , Processing time and status .

 

 

 

among flink Of savepoint Is a consistent state mirror , Multiple application statuses can be recorded , So that applications can be upgraded and expanded safely , You can also open multiple applications , To complete A/B test .

1.2 Data analysis application

What is a data analysis application ：

Streaming data analysis application ：

  Write the results to an external database , Or in the internal state

  then , Report and other data analysis applications , From an external database , Or internal state reading ：

Typical data analysis scenarios ：

Advantages of streaming query ：

1. Real time data analysis , Relative batch offline data analysis , Low latency

2. It solves the boundary problem in batch off-line data processing

3. Use the fault recovery mechanism , It solves the complex scheduling problem of batch offline jobs （ And upstream operation failure caused by upstream and downstream dependency , Downstream jobs cannot run ）

 

 

 1.2.1 flink How to support data analysis applications

1. Provide standards SQL Interface , Whether it's querying static datasets , Or real-time data sets .

It also supports customization UDF function .

and , If you need to further customize the processing logic , You can also use DataSetAPI and DataStreamAPI Provide a lower level method

in addition ,Flink Of Gelly Library provides more libraries for graph calculation .

 

 

 

 

1.3 Data pipeline application

Introduction to assembly line ： be relative to ETL Homework , Pipeline applications are more real-time

 

And can realize the directory , Real time monitoring of documents, etc , Once changes occur, they can be captured at any time .

 

Typical scenarios for pipeline applications ：

Advantages of assembly line ：

 

flink How to support pipeline application scenarios ：

Provide a large number of connectors ：

Support time partition , To write files