Specify flume introduction, installation and configuration

Hello everyone , I meet you again , I'm the king of the whole stack .

editorial ：

This paper summarizes “Hadoop The ecological system ” One of them ——Apache Flume

Write in the first two ：

Description of the software used ：

One 、 What is? Apache Flume

Official website ：Flume is a distributed, reliable, and availableservicefor efficientlycollecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

1、flume It is a distributed log collection system , With high reliability 、 High availability 、 Business management 、 Failed restart and other functions . Data processing speed is fast , It can be completely used in the production environment . Frequently used distributed log collection system ： Apache Flume、 Facebook Scribe、 Apache Chukwa 2、flume The core is agent.

agent It's a java process , Execute on the log collection side , adopt agent Receiving log . Then store it temporarily , And send it to the destination . 3、agent includes 3 Core components ：source、channel、sink.

3.1、source The component is dedicated to collecting logs . It can handle various types and formats of log data , contain avro、thrift、exec、jms、spooling directory、netcat、 sequence generator、syslog、http、legacy、 Define your own .source After the component collects the data . Temporarily stored in channel in .

3.2、channel The component is in the agent Designed for temporary storage of data , Can be stored in memory、jdbc、file、 Define your own . channel The data in is only in sink It will not be deleted until it is successfully sent . 3.3、sink A component is a component used to send data to a destination , The destination contains hdfs、logger、avro、thrift、ipc、file、null、hbase、solr、 Define your own . 4、 In the whole process of data transmission . What's flowing is event. The transaction guarantee is in event Level . 5、flume It can support multiple levels flume Of agent, Support fan in (fan-in)、 Fan out (fan-out).

Flume Simple processing diagram

Flume Schematic diagram of complex processing

Two 、Flume Installation configuration

2.1、 download Flume

Flume Download link

2.2、 decompression

cd /home/yujianxin/flume tar -zxvf apache-flume-1.4.0-src.tar.gz ; tar -zxvfapache-flume-1.4.0-bin.tar.gz

And then put src All contents in the directory are copied to bin Catalog

2.3、 To configure Flume environment variable

3、 ... and 、 verification Flume Is the installation successful

OK！

The screenshot above appears , explain Flume Installation successful .

Copyright notice ： This article is an original blog article , Blog , Without consent , Shall not be reproduced .

Publisher ： Full stack programmer stack length , Reprint please indicate the source ：https://javaforall.cn/117390.html Link to the original text ：https://javaforall.cn