Flume learning III

Flume Business

 flume Streaming processing has two transactions Put Business and Take Business
        Put The flow of a transaction is ,source And channel Communicating yes ,doput Write data to temporary buffer putlist, It's triggering doCommit Submitted to the channel, if channel Full or otherwise abnormal , It will trigger doRollback Roll back to source Waiting for resubmission .
        Take Business ：sink And channel communicate , Put the data into Takelist Inside ,sink Again from Talelist Pull data from it , If an exception occurs during data transmission , The trigger rollback take takelist Data rollback to channel Memory queue . 

 

 Flume Agent internals

Important components ：
ChannelSelector The purpose of this is to select Event Which one will be sent to Channel. There are two types of them , Namely Replicating（ Copy ） and Multiplexing（ Multiplexing ）.
ReplicatingSelector Will be the same Event To all Channel,Multiplexing According to the phase Principle of due , Will be different Event To different Channel.

SinkProcessor common Yes 3、 ... and Kind of class type , branch other yes DefaultSinkProcessor 、 LoadBalancingSinkProcessor and FailoverSinkProcessor DefaultSinkProcessor Yes Should be Of yes single individual Of Sink , LoadBalancingSinkProcessor and FailoverSinkProcessor The corresponding is Sink Group,LoadBalancingSinkProcessor Negative... Can be achieved Load balancing function ,FailoverSinkProcessor Can realize the function of fail over .

Flume topology  

Simple series connection ：

This pattern is to put more than one flume The sequence is connected , From the initial source From the beginning to the end sink The transfer of Destination storage system . This mode is not recommended to bridge too many flume Number ,flume Too much will not only affect the transmission rate , And once a node is in transit flume Downtime , It will affect the whole transmission system .

Replication and multiplexing ：

 

 Flume Support the flow of events to one or more destinations . This mode can copy the same data to multiple channel in , Or distribute different data to different channel in ,sink You can choose to send to different destinations The earth .

Load balancing and fail over ：

 Flume Support the use of multiple sink Logically, there is a sink Group ,sink Group with different SinkProcessor Can achieve load balancing and error recovery function .

polymerization ：

  This pattern is our most common , It's also very practical , daily web Applications are usually distributed on hundreds of servers , The great Even thousands of 、 Tens of thousands of servers . Generated log , It's also very troublesome to deal with . use flume This combination of It can solve this problem very well , One for each server flume Collect logs , Transfer to a centralized log collector flume, And so on flume Upload to hdfs、hive、hbase etc. , Log analysis .