Flume common commands and basic operations

flume There are three major categories of components ：

1. source  Components , specify data source
2. channel  Components , As a cache , Connect source and sink
3. sink  Components , Specify the data destination

Case study 1： Use flume Acquisition from 44444 Data from port

1） Ready to configure

#  Declare three components  a1 = agent1
a1.sources = r1
a1.channels = c1
a1.sinks = k1

#  Definition source Information 
# r1 type , What kind of data to monitor 
a1.sources.r1.type = netcat
#  monitor ip
a1.sources.r1.bind = 192.168.88.161
#  Port number 
a1.sources.r1.port = 44444
# r1 Connect to c1
a1.sources.r1.channels = c1

#  Definition channel Information 
a1.channels.c1.type = memory

#  Definition sink Information 
#  Set the type   Print... In the terminal 
a1.sinks.k1.type = logger
#  To which channel
a1.sinks.k1.channel = c1

2） Write configuration

1） open flume The installation directory

cd /export/server/apache-flume-1.9.0-bin/conf

2） Create a profile

touch intro_source_netcat_sink_logger.conf

3） hold Top configuration Write to file （ Remember to delete the Chinese Notes ）

vim intro_source_netcat_sink_logger.conf

3） start-up Flume

flume-ng agent -f /export/server/apache-flume-1.9.0-bin/conf/intro_source_netcat_sink_logger.conf -n a1 -Dflume.root.logger=INFO,console

4） towards 44444 Port send data

#  install telnet
yum install telnet

#  towards 44444 Port send data 
telnet 192.168.88.161 44444

---

Case study 2： Use flume Acquisition from kafka The data of

1） Ready to configure

a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile =  /export/data/momo_data/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 =  /export/data/momo_data/MOMO_DATA.dat

a1.channels.c1.type = memory

a1.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers=node1:9092,node2:9092,node3:9092
a1.sinks.k1.kafka.topic=MOMO_MSG
a1.sinks.k1.channel=c1

2） Write configuration

1） open flume The installation directory

cd /export/server/apache-flume-1.9.0-bin/conf

2） Create a profile

touch intro_source_netcat_sink_logger.conf

3） hold Top configuration Write to file （ Remember to delete the Chinese Notes ）

vim intro_source_netcat_sink_logger.conf

3） Use flume Acquisition from kafka The data of

1） start-up kafka

nohup kafka-server-start.sh /export/server/kafka_2.12-2.4.1/config/server.properties&

2） establish topics

kafka-topics.sh --create --bootstrap-server node1.itcast.cn:9092 --topic MOMO_MSG --partitions 3 --replication-factor 2

3） start-up flume

flume-ng agent -f /export/server/apache-flume-1.9.0-bin/conf/source_taildir_sink_kafka.conf -n a1 -Dflume.root.logger=INFO,console

4） start-up kafka consumer Through this, the consumer checks that the data is actually written to this MOMO_MSG

kafka-console-consumer.sh --bootstrap-server node1.itcast.cn:9092 --topic MOMO_MSG

5） Start production data jar

cd /export/data/momo_init/
java -jar MoMo_DataGen.jar MoMo_Data.xlsx /export/data/momo_data/ 1000