当前位置:网站首页>01-spooldir
01-spooldir
2022-07-02 10:31:00 【Lucky lucky】
-------------------------------------- spooldir -----------------------------------------
1、 establish cnf
vim flume_spooldir_file_hdfs.cnf
2、 Configuration information
# Component description
a1.sources = s1
a1.channels = c1
a1.sinks = k1
# Initialize the data source
a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /root/data/flume
a1.sources.s1.ignorePattern = ^(.)*\\.bak$
a1.sources.s1.fileSuffix = .bak
# Initialize channel
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /opt/software/flume190/mydata/checkpoint
a1.channels.c1.dataDirs = /opt/software/flume190/mydata/data
a1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 10000
# Initialize data slot
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://192.168.19.130:9820/flume/events/fakeorder/%Y-%m-%d/%H
a1.sinks.k1.hdfs.round = true # Time abandonment
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.filePrefix = log_%Y%m%d_%H
a1.sinks.k1.hdfs.fileSuffix = .log
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.bacthSize = 1000
a1.sinks.k1.hdfs.threadsPoolSize = 4
a1.sinks.k1.hdfs.idleTimeout = 0
a1.sinks.k1.hdfs.minBlockReplicas = 1
# Associated components
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
3、 start-up
flume-ng agent -n a1 -c conf/ -f flume-conf-files/flume_spooldir_file_hdfs.cnf -Dflume.root.logger=INFO,console
边栏推荐
- 【Lua】常见知识点汇总(包含常见面试考点)
- Blender volume fog
- ICLR 2022: how does AI recognize "things I haven't seen"?
- ue4材质的入门和原理笔记
- [IDL] Research
- Sum the two numbers to find the target value
- 【避坑指南】Unity3D项目接入腾讯Bugly工具时遇到的坑
- [200 Shengxin literatures] 95 multiomics exploration TNBC
- UE4 night lighting notes
- pytest--之测试报告allure配置
猜你喜欢
What is the relationship between realizing page watermarking and mutationobserver?
MPLS experiment
flink 提交程序
虚幻AI蓝图基础笔记(万字整理)
Blender石头雕刻
Following nym, the new project Galaxy token announced by coinlist is gal
Ue5 - AI pursuit (blueprint, behavior tree)
Pytest-- test report allure configuration
AutoCAD - layer Linetype
A model can do two things: image annotation and image reading Q & A. VQA accuracy is close to human level | demo can be played
随机推荐
Brief analysis of edgedb architecture
Leetcode -- the nearest common ancestor of 236 binary tree
ue虛幻引擎程序化植物生成器設置——如何快速生成大片森林
Basic notes of illusory AI blueprint (10000 words)
2021-10-04
【Lua】常见知识点汇总(包含常见面试考点)
Unreal material editor foundation - how to connect a basic material
两数之和,求目标值
【避坑指南】使用UGUI遇到的坑:Text组件无法首行缩进两格
[ue5] two implementation methods of AI random roaming blueprint (role blueprint and behavior tree)
2021-10-04
ue虚幻引擎程序化植物生成器设置——如何快速生成大片森林
How much is it to develop a system software in Beijing, and what funds are needed to develop the software
What is call / cc- What is call/cc?
lunix重新分配root 和 home 空间内存
07 data import sqoop
Following nym, the new project Galaxy token announced by coinlist is gal
Project practice, redis cluster technology learning (16)
Mock Server基本使用方法
快速做出原型