当前位置:网站首页>Specify flume introduction, installation and configuration
Specify flume introduction, installation and configuration
2022-07-06 18:30:00 【Full stack programmer webmaster】
Hello everyone , I meet you again , I'm the king of the whole stack .
editorial :
This paper summarizes “Hadoop The ecological system ” One of them ——Apache Flume
Write in the first two :
Description of the software used :
One 、 What is? Apache Flume
Official website :Flume is a distributed, reliable, and availableservicefor efficientlycollecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
1、flume It is a distributed log collection system , With high reliability 、 High availability 、 Business management 、 Failed restart and other functions . Data processing speed is fast , It can be completely used in the production environment . Frequently used distributed log collection system : Apache Flume、 Facebook Scribe、 Apache Chukwa 2、flume The core is agent.
agent It's a java process , Execute on the log collection side , adopt agent Receiving log . Then store it temporarily , And send it to the destination . 3、agent includes 3 Core components :source、channel、sink.
3.1、source The component is dedicated to collecting logs . It can handle various types and formats of log data , contain avro、thrift、exec、jms、spooling directory、netcat、 sequence generator、syslog、http、legacy、 Define your own .source After the component collects the data . Temporarily stored in channel in .
3.2、channel The component is in the agent Designed for temporary storage of data , Can be stored in memory、jdbc、file、 Define your own . channel The data in is only in sink It will not be deleted until it is successfully sent . 3.3、sink A component is a component used to send data to a destination , The destination contains hdfs、logger、avro、thrift、ipc、file、null、hbase、solr、 Define your own . 4、 In the whole process of data transmission . What's flowing is event. The transaction guarantee is in event Level . 5、flume It can support multiple levels flume Of agent, Support fan in (fan-in)、 Fan out (fan-out).
Flume Simple processing diagram
Flume Schematic diagram of complex processing
Two 、Flume Installation configuration
2.1、 download Flume
2.2、 decompression
cd /home/yujianxin/flume tar -zxvf apache-flume-1.4.0-src.tar.gz ; tar -zxvfapache-flume-1.4.0-bin.tar.gz
And then put src All contents in the directory are copied to bin Catalog
2.3、 To configure Flume environment variable
3、 ... and 、 verification Flume Is the installation successful
OK!
The screenshot above appears , explain Flume Installation successful .
Copyright notice : This article is an original blog article , Blog , Without consent , Shall not be reproduced .
Publisher : Full stack programmer stack length , Reprint please indicate the source :https://javaforall.cn/117390.html Link to the original text :https://javaforall.cn
边栏推荐
猜你喜欢
随机推荐
AFNetworking框架_上传文件或图像server
CSRF漏洞分析
Echart simple component packaging
std::true_type和std::false_type
HMS core machine learning service creates a new "sound" state of simultaneous interpreting translation, and AI makes international exchanges smoother
随着MapReduce job实现去加重,多种输出文件夹
阿里云国际版ECS云服务器无法登录宝塔面板控制台
關於這次通信故障,我想多說幾句…
首先看K一个难看的数字
2022 Summer Project Training (II)
【Swoole系列2.1】先把Swoole跑起来
Redis的五种数据结构
Markdown syntax for document editing (typera)
C语言自动预订飞机票问题
传输层 拥塞控制-慢开始和拥塞避免 快重传 快恢复
Recursive way
Tree-LSTM的一些理解以及DGL代码实现
Jielizhi obtains the currently used dial information [chapter]
2019阿里集群数据集使用总结
Insert dial file of Jerry's watch [chapter]