当前位置:网站首页>Flume common commands and basic operations
Flume common commands and basic operations
2022-07-05 12:33:00 【JKing_ one hundred and sixty-eight】
flume There are three major categories of components :
source Components , specify data source channel Components , As a cache , Connect source and sink sink Components , Specify the data destination
Case study 1: Use flume Acquisition from 44444 Data from port
1) Ready to configure
# Declare three components a1 = agent1
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Definition source Information
# r1 type , What kind of data to monitor
a1.sources.r1.type = netcat
# monitor ip
a1.sources.r1.bind = 192.168.88.161
# Port number
a1.sources.r1.port = 44444
# r1 Connect to c1
a1.sources.r1.channels = c1
# Definition channel Information
a1.channels.c1.type = memory
# Definition sink Information
# Set the type Print... In the terminal
a1.sinks.k1.type = logger
# To which channel
a1.sinks.k1.channel = c1
2) Write configuration
1) open flume The installation directorycd /export/server/apache-flume-1.9.0-bin/conf
2) Create a profiletouch intro_source_netcat_sink_logger.conf
3) hold Top configuration Write to file ( Remember to delete the Chinese Notes )vim intro_source_netcat_sink_logger.conf
3) start-up Flume
flume-ng agent -f /export/server/apache-flume-1.9.0-bin/conf/intro_source_netcat_sink_logger.conf -n a1 -Dflume.root.logger=INFO,console
4) towards 44444 Port send data
# install telnet
yum install telnet
# towards 44444 Port send data
telnet 192.168.88.161 44444
Case study 2: Use flume Acquisition from kafka The data of
1) Ready to configure
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile = /export/data/momo_data/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /export/data/momo_data/MOMO_DATA.dat
a1.channels.c1.type = memory
a1.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers=node1:9092,node2:9092,node3:9092
a1.sinks.k1.kafka.topic=MOMO_MSG
a1.sinks.k1.channel=c1
2) Write configuration
1) open flume The installation directorycd /export/server/apache-flume-1.9.0-bin/conf
2) Create a profiletouch intro_source_netcat_sink_logger.conf
3) hold Top configuration Write to file ( Remember to delete the Chinese Notes )vim intro_source_netcat_sink_logger.conf
3) Use flume Acquisition from kafka The data of
1) start-up kafkanohup kafka-server-start.sh /export/server/kafka_2.12-2.4.1/config/server.properties&
2) establish topicskafka-topics.sh --create --bootstrap-server node1.itcast.cn:9092 --topic MOMO_MSG --partitions 3 --replication-factor 2
3) start-up flumeflume-ng agent -f /export/server/apache-flume-1.9.0-bin/conf/source_taildir_sink_kafka.conf -n a1 -Dflume.root.logger=INFO,console
4) start-up kafka consumer Through this, the consumer checks that the data is actually written to this MOMO_MSGkafka-console-consumer.sh --bootstrap-server node1.itcast.cn:9092 --topic MOMO_MSG
5) Start production data jarcd /export/data/momo_init/ java -jar MoMo_DataGen.jar MoMo_Data.xlsx /export/data/momo_data/ 1000
边栏推荐
- The evolution of mobile cross platform technology
- Distributed cache architecture - cache avalanche & penetration & hit rate
- Swift - enables textview to be highly adaptive
- Resnet+attention project complete code learning
- One article tells the latest and complete learning materials of flutter
- Pytorch two-layer loop to realize the segmentation of large pictures
- byte2String、string2Byte
- Matlab struct function (structure array)
- MySQL installation, Windows version
- Experimental design - using stack to realize calculator
猜你喜欢
Get the variable address of structure member in C language
Anaconda creates a virtual environment and installs pytorch
Flutter2 heavy release supports web and desktop applications
Course design of compilation principle --- formula calculator (a simple calculator with interface developed based on QT)
A guide to threaded and asynchronous UI development in the "quick start fluent Development Series tutorials"
Seven ways to achieve vertical centering
Keras implements verification code identification
Average lookup length when hash table lookup fails
[figure neural network] GNN from entry to mastery
Reinforcement learning - learning notes 3 | strategic learning
随机推荐
UNIX socket advanced learning diary - advanced i/o functions
Pytoch uses torchnet Classerrormeter in meter
PIP command reports an error pip is configured with locations that requires tls/ssl problems
C language structure is initialized as a function parameter
Distributed solution - completely solve website cross domain requests
ZABBIX ODBC database monitoring
Want to ask, how to choose a securities firm? Is it safe to open an account online?
mysql拆分字符串做条件查询
II. Data type
The relationship between the size change of characteristic graph and various parameters before and after DL convolution operation
What is the difference between canvas and SVG?
Pytoch implements tf Functions of the gather() function
Understand kotlin from the perspective of an architect
Two minutes will take you to quickly master the project structure, resources, dependencies and localization of flutter
Simple production of wechat applet cloud development authorization login
GPON technical standard analysis I
Knowledge representation (KR)
Hexadecimal conversion summary
Matlab superpixels function (2D super pixel over segmentation of image)
Constructing expression binary tree with prefix expression