Filebeat collects logs to elk

1、 install

Official download of the latest version , Upload server decompression

tar -zxvf filebeat-7.17.4-linux-x86_64.tar.gz

2、 Print to console debugging

1. To configure stdout.yml

#  Input 
filebeat.inputs:
  #  The standard input 
- type: stdin
  enabled: true
#  Output 
#  Output to console 
output.console:
  pretty: true
  enable: true

2、 Start command

./filebeat -e -c stdout.yml

3、 Collect to Elasticsearch

1） To configure filebeat.yml

filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/access*.log
# filebeat collect java Multi line log 
# multiline.pattern: ^\[ # Because the beginning of the log is all kinds of dates , Use this to match 
# multiline.negate: true # Turn on multiline mode 
# multiline.match: after
  tags: ["access"]
  encoding: utf-8
  prospector.scanner.exclude_files: ['.gz$']
  fields_under_root: true
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: message
## system log 
- type: filestream
  id: sys-log
  enabled: false
  paths:
    - /var/log/syslog*.log
  tags: ["sys-log"]
  encoding: utf-8
  prospector.scanner.exclude_files: ['.gz$']
# fields_under_root: true

#json.keys_under_root: true #  Turn on json Format  
#json.overwrite_keys: true
setup.ilm.enabled: false
setup.template.enabled: false #  No module version , Just use what I configured here 
setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "${filebeat_user}"
  password: "${filebeat_passwd}"
  indices:
    - index: "filebeat-access-%{+yyyy.MM.dd}"
      when.contains: # adopt when Judge , When the tag is a, the corresponding index is created  
         tags: "access"
    - index: "filebeat-sys-log-%{+yyyy.MM.dd}"
      when.contains:
         tags: "sys-log"

# setup.template.enabled: false
# setup.template.name: "filebeat"
# setup.template.pattern: "filebeat-*"
# setup.template.overwrite: true
#nginx  To configure  
# setup.template.enabled: false
# setup.template.name: "nginx"
# setup.template.pattern: "nginx-*"
# setup.template.overwrite: true

4、 Boot up configuration

1. Normal start

nohup ./filebeat -e -c xxxx.yml >> filebeat.log &

or

nohup ./filebeat -c ./filebeat.yml -e > /dev/null 2>&1 &

2. Boot configuration （CentOS 7.X）

• New service

vi /usr/lib/systemd/system/filebeat.service

• Edit document

[Unit]
Description=filebeat
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml
Restart=always  # Set to automatic restart after dropping the line , The process will restart automatically after being forced to kill 
[Install]
WantedBy=multi-user.target

• Start the service

systemctl start filebeat.service
systemctl enable filebeat.service

systemctl daemon-reload  　　　　　　　　   # Load the configuration 
systemctl enable filebeat             　　# Set power on self start 
systemctl disable filebeat            　　# Stop power on and start up 

systemctl start filebeat              　　# start-up filebeat service 
systemctl restart filebeat　          　　# Restart the service 

systemctl status filebeat             　　# View the current status of the service 
systemctl list-units --type=service      # View all started services 

5、keystore Use

be used for filebeat.yml Get parameters dynamically in

# Signed documents 
filebeat keystore create
# New password 
filebeat keystore add ES_PWD
# Update signature file 
filebeat keystore add ES_PWD --force
# Signed documents   list 
filebeat keystore list
# Remove signature file 
filebeat keystore remove ES_PWD

6、 Pit encountered

Detected. log The change of , but kibana and ES No corresponding index file can be found on .

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://x.x.x.x:9200)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 405 Method Not Allowed:

[elasticsearch] elasticsearch/client.go:414     Cannot index event publisher.Event{
    Content:beat.Event{
    Timestamp:time.Date(2022, time.June, 8, 18, 8, 9, 18041600, time.Local), Meta:null,Cache:publisher.EventCache{
    m:common.MapStr(nil)}} (status=404): {
    "type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([.security,.security-6,.monitoring-*,.watch*,.triggered_watches,.quota]) doesn't match","index_uuid":"_na_","index":"logstash-ld456-access-2022.06.08"}, dropping event!

The solution is to turn on automatic creation index Configuration of , Or manually create an index based on the error information

PUT /_cluster/settings
{
    
    "persistent" : {
    
        "action": {
    
          "auto_create_index": "true"
        }
    }
}

7、 Keep only message news , And remove the other fields

Mode one

processors:
  - decode_json_fields:
        fields: ["message"]
        target: ""
  - drop_fields:
        fields: ["ecs","cloud","host","agent"]
        ignore_missing: true

Mode two

filebeat.inputs:
- type: filestream
  ...
  parsers:
    - ndjson:
        target: ""
        message_key: msg
    - multiline:
        type: counter
        lines_count: 3

Official references ：https://www.elastic.co/guide/en/beats/filebeat/7.17/elasticsearch-output.html