Flume learning 1

Definition ：

flume Is a highly available 、 Highly reliable 、 A distributed , Log collection 、 polymerization 、 The transmission system . Only log type data , Data such as pictures and videos are not good .

effect ：

flume The function of is to read data from the local disk and then transfer the data to HDFS Do offline analysis

flume framework ： Data sources --->flume---> Where the data goes

flume Components ：

source：Source Is responsible for receiving data to Flume Agent The components of .Source Components can handle various types 、 Various Log data in format , Include avro、thrift、exec、jms、spooling directory、netcat、sequence generator、syslog、http、legacy.(flume To accept from wedServer Read data , Dealing with the outside world . Reading documents 、 Port data, etc ) avro For multi tier flume

channel： The function is to play a buffer role , Don't store data , When all data is sent to sink,channel The data will be deleted

sink：Sink Keep polling Channel And remove them in batches , And batch write these events to the storage Or indexing system 、 Or it could be sent to another Flume Agent. Sink Component destinations include hdfs、logger、avro、thrift、ipc、file、HBase、solr、 Self determination The righteous .

agent：Agent It's a JVM process , It sends data from source to destination in the form of events . Agent There are mainly 3 Component composition ,Source、Channel、Sink.

        

Flume Installation and deployment

flume Official address ：https://flume.apache.org/

Document address ：https://flume.apache.org/FlumeUserGuide.html

flume Download address ：http://archive.apache.org/dist/flume/

After downloading , take apache-flume-1.7.0-bin.tar.gz Upload to linux
Re decompress ：

tar -zxf apache-flume-1.7.0-bin.tar.gz

You can decompress the apache-flume-1.7.0-bin Renamed as flume

 mv apache-flume-1.7.0-bin flume

  take flume/conf Under the flume-env.sh.template Change the file to flume-env.sh, And configuration flumeenv.sh file

 mv flume-env.sh.template flume-env.sh

 vi flume-env.sh

stay flume-env.sh Add... To the file   export JAVA_HOME= you linux In the virtual machine jdk The path of
have access to echo $JAVA_HOME see jdk The path of

such flume The installation and deployment of is completed ！