当前位置:网站首页>Journal d'acquisition du faisceau de fichiers à Elk
Journal d'acquisition du faisceau de fichiers à Elk
2022-06-13 08:22:00 【Soleil glacé】
1、Installation
Téléchargement officiel de la dernière version,Télécharger le serveur décompresser
tar -zxvf filebeat-7.17.4-linux-x86_64.tar.gz
2、Imprimer sur la console pour déboguer
- Configurationstdout.yml
# Entrée
filebeat.inputs:
# Entrée standard
- type: stdin
enabled: true
# Produits
# Sortie vers la console
output.console:
pretty: true
enable: true
2、Commande de démarrage
./filebeat -e -c stdout.yml
3、AcquisElasticsearch
1)Configurationfilebeat.yml
filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/access*.log
# filebeatCollectejava Journal multiligne
# multiline.pattern: ^\[ # Parce que les journaux commencent par des dates différentes , Utilisez ceci pour faire correspondre
# multiline.negate: true # Activer le mode multiligne
# multiline.match: after
tags: ["access"]
encoding: utf-8
prospector.scanner.exclude_files: ['.gz$']
fields_under_root: true
json.keys_under_root: true
json.add_error_key: true
json.message_key: message
##Journal du système
- type: filestream
id: sys-log
enabled: false
paths:
- /var/log/syslog*.log
tags: ["sys-log"]
encoding: utf-8
prospector.scanner.exclude_files: ['.gz$']
# fields_under_root: true
#json.keys_under_root: true # OuvertjsonFormat
#json.overwrite_keys: true
setup.ilm.enabled: false
setup.template.enabled: false # Pas de version modulaire , Seulement ce que j'ai configuré ici
setup.template.settings:
index.number_of_shards: 3
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "${filebeat_user}"
password: "${filebeat_passwd}"
indices:
- index: "filebeat-access-%{+yyyy.MM.dd}"
when.contains: #AdoptionwhenPorter un jugement, Créez l'index approprié lorsque l'étiquette est quelque chose
tags: "access"
- index: "filebeat-sys-log-%{+yyyy.MM.dd}"
when.contains:
tags: "sys-log"
# setup.template.enabled: false
# setup.template.name: "filebeat"
# setup.template.pattern: "filebeat-*"
# setup.template.overwrite: true
#nginx Configuration
# setup.template.enabled: false
# setup.template.name: "nginx"
# setup.template.pattern: "nginx-*"
# setup.template.overwrite: true
4、 Configuration de démarrage
- Démarrage normal
nohup ./filebeat -e -c xxxx.yml >> filebeat.log &
or
nohup ./filebeat -c ./filebeat.yml -e > /dev/null 2>&1 &
- Configuration de démarrage automatique(CentOS 7.X)
- Nouveau service
vi /usr/lib/systemd/system/filebeat.service
- Modifier le document
[Unit]
Description=filebeat
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml
Restart=always # Mise en place d'un redémarrage automatique en cas de déconnexion , Le processus redémarre automatiquement après un meurtre forcé
[Install]
WantedBy=multi-user.target
- Démarrer le service
systemctl start filebeat.service
systemctl enable filebeat.service
systemctl daemon-reload #Configuration de chargement
systemctl enable filebeat #Réglage de l'auto - démarrage
systemctl disable filebeat #Arrêt du démarrage automatique
systemctl start filebeat #DémarragefilebeatServices
systemctl restart filebeat #Redémarrer le service
systemctl status filebeat #Voir l'état actuel du service
systemctl list-units --type=service #Voir tous les services démarrés
5、keystoreUtiliser
Pourfilebeat.yml Paramètres d'acquisition dynamique dans
#Fichier de signature
filebeat keystore create
# Nouveau mot de passe
filebeat keystore add ES_PWD
# Mettre à jour le fichier de signature
filebeat keystore add ES_PWD --force
#Fichier de signature Liste
filebeat keystore list
# Supprimer le fichier de signature
filebeat keystore remove ES_PWD
6、Fosse rencontrée
DétectélogChangement de,Mais...kibanaEtES Aucun fichier d'index correspondant n'a été trouvé sur .
ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://x.x.x.x:9200)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 405 Method Not Allowed:
[elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{
Content:beat.Event{
Timestamp:time.Date(2022, time.June, 8, 18, 8, 9, 18041600, time.Local), Meta:null,Cache:publisher.EventCache{
m:common.MapStr(nil)}} (status=404): {
"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([.security,.security-6,.monitoring-*,.watch*,.triggered_watches,.quota]) doesn't match","index_uuid":"_na_","index":"logstash-ld456-access-2022.06.08"}, dropping event!
La solution est d'activer la création automatique indexConfiguration de, Ou créer manuellement un index basé sur les informations d'erreur
PUT /_cluster/settings
{
"persistent" : {
"action": {
"auto_create_index": "true"
}
}
}
7、Réservé uniquementmessageMessage, Et supprimer les autres champs
Mode 1
processors:
- decode_json_fields:
fields: ["message"]
target: ""
- drop_fields:
fields: ["ecs","cloud","host","agent"]
ignore_missing: true
Mode 2
filebeat.inputs:
- type: filestream
...
parsers:
- ndjson:
target: ""
message_key: msg
- multiline:
type: counter
lines_count: 3
Documents de référence officiels:https://www.elastic.co/guide/en/beats/filebeat/7.17/elasticsearch-output.html
边栏推荐
- Rust writes near smart contract
- Disk C is full? A few simple tips teach you to release and clean up tens of gigabytes of space on the C disk, the most effective way to clean up the C disk
- [notes] like the solution to the problem of slow query (index + explicitly specifying query fields)
- Motiko basic syntax in dfinity (ICP) -8
- 微服务项目搭建二:数据库设计
- CCNP_ Bt-ospf big experiment (1)
- Dfinity (ICP) basic development tutorial-5
- 【博弈论-完全信息静态博弈】 Nash均衡的应用
- Dfinity (ICP) development problems and solutions-6
- Shell脚本常用开发规范
猜你喜欢
ERP basic data concept
Format_ String_ Server
MySQL queries difference sets (missing data) by linking tables based on an associated field
How to modify desktop path in win10 system
微服务系统架构搭建一:环境搭建
Microservice project construction III: automatic code generation
Gtk+ programming example on page 115 - simplest progress bar 2 with steps to write GTK program using anjuta
When submitting the laravel admin form and using the required verification, an error is reported when the value is 0
酒水批发行业应当如何高效管理商品与库存
将solidworks建的机器人模型导入到ros中
随机推荐
Dfinity (ICP) deployment and development-2
母婴用品批发行业使用管理软件提高效率 实现降本增效
学习记录4: einops // cudnn.benchamark=true // hook
字符串的逆序与比较
淘宝商品销量接口/淘宝商品销量监控接口/商品累计销量接口
Wechat upload picture material interface
Is signed or unsigned selected to create an integer field in MySQL? The answer is as follows:
File upload question type
Overview of cross chain protocol IBC
有什么好的管理软件来解决茶叶批发商面临的难题
DNS domain name resolution service
Edge browser uses bdtab new tab plug-in (BD new tab)
[notes] like the solution to the problem of slow query (index + explicitly specifying query fields)
CCNP_ BT-MGRE
Why do wholesalers use the order system
CCNP_ Bt-ospf big experiment (1)
【博弈论-完全信息静态博弈】 Nash均衡的应用
Founder of Starbucks: no longer open "public toilets" to non store consumers for safety reasons
About redis encapsulation tool class using distributed locks
Mongodb test case