Flume learning notes

  function

Distributed real-time files 、 Network port data flow collection , Data from various data sources can be collected to various destinations in real time

characteristic

Real time acquisition Monitor data sources in real time , Collect data as soon as it is generated

Comprehensive function The common data sources and destinations of big data are encapsulated with corresponding interfaces

Allow custom development Java Source code of development , It provides an interface for user-defined development

Development is relatively simple Develop a configuration file , Just write the configuration

It can realize distributed collection Itself is not a distributed tool , It can realize distributed collection

framework

Agent： One flume The program is a Agent

Event：flume The collected data is encapsulated as Event Object to transmit

Source： Monitor data sources in real time , As soon as the data source generates data, it collects

Channel： Be responsible for temporarily storing the collected data , Will all Event Temporary storage

Sink： Responsible for Channel Send the data in to the destination , Initiative from Channel Count

Multi data source architecture

 

 

Design purpose ： Write a copy to different destinations

Multi tier architecture

 

Design purpose ： Prevent multiple Flume The program directly interacts with the destination , Affect destination performance

Usage mode

offline （ Collect to HDFS）： To configure Source and Sink file , start-up Hive and HDFS, Submit and run on the command line

real time （ Collect to Kafka）： To configure Source and Sink file , Collect to kafka, For consumption by real-time computing programs

Comparison of similar software

Sqoop The bottom is MapReduce, It is suitable for collection with a large amount of offline data

Flume Suitable for real-time collection of files , Network port

Canal Suitable for real-time acquisition MySQL database

Comparison of similar software

sqoop The bottom is MapReduce, It is suitable for collection with a large amount of offline data

Flume Suitable for real-time collection of files , Network port

Canal Suitable for real-time acquisition MySQL database

Advanced components

Interceptor： Interceptor , stay source Convert each piece of data into event When , Can be in event Head shop addition kv Or filter the data

Add data ：

Timestamp Interceptor

Host Interceptor

Static Interceptor

Filtering data ：

Regex Filtering Interceptor

Channel Select

Default source Data is sent to each channel One copy , According to Agent The head of the key Different , Send to different channel

Sink Processor

function

Load balancing   

Multiple sink With sink group Way to work together , One of the faults , It can also collect normally

Fail over

Multiple sink, One job , Others don't work , Only at work sink It works only after failure , Ensure the normal collection