Filebeat collects log data and transfers it to redis. Different es indexes are created based on log fields through logstash

1.Filebeat.yml To configure

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  exclude_files: ['.gz$','INFO']
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["nginx-log-messages"]
  fields:
    log_source: messages
  fields_under_root: true


output.redis:
  hosts: ["192.168.0.111:6379"]
  key: nginx_log
  password: nginxredis
  db: 0

Copy

Parameter description

  fields:
    log_source: messages
  fields_under_root: true

Copy

  Use fields It means that filebeat Add one more field to the collected log log_source, Its value is messages, Used in logstash Of output Output to elasticsearch Determine the source of the log , So as to establish the corresponding index if fields_under_root Set to true, Indicates that the newly added field above is a top-level parameter .

Top level fields in output Output to elasticsearch Use the following in ：

[[email protected] logstash]# vim config/logstash.conf
input {
  redis {
    data_type => "list"
    host => "192.168.0.111"
    db => "0"
    port => "6379"
    key => "nginx_log"
    password => "nginxredis"
  }
}

output {
  #  according to redis key  messages_secure  In the corresponding list value , One of the parameters of each row of data to determine the log source 
  if [log_source] == 'messages' {  #  Pay attention to the writing of judgment conditions 
    elasticsearch {
      hosts => ["192.168.0.111:9200"]
      index => "nginx-message-%{+YYYY.MM.dd}"
      #user => "elastic"
      #password => "elastic123"
    }
  }

  # Or it can be based on tags Judge 
  if "nginx-log-messages" in [tags] {
    elasticsearch {
        hosts => [""192.168.0.111:9200"]
        index => "nginx-message-%{+YYYY.MM.dd}"
    }
  }

} 

Copy

2. Logs of multiple applications are output to redis

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  tags: ["nginx-log-access"]
  fields:
    log_source: access
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/error.log
  tags: ["nginx-log-error"]
  fields:
    log_source: error
  fields_under_root: true

output.redis:
  hosts: ["192.168.0.111:6379"]
  key: nginx_log
  password: nginxredis
  db: 0

Copy

  stay redis The effect shown in is that it will be output to key value nginx_log In the corresponding list , according to key Values are indistinguishable , Only according to the key In each row of data in the value list log_source Or use the attributes defined by yourself to determine which application log this line is .

3. Different application logs use different rediskey value

Use output.redis Medium keys value , The official example

output.redis:
  hosts: ["localhost"]
  key: "default_list"
  keys:
    - key: "error_list"   # send to info_list if `message` field contains INFO
      when.contains:
        message: "error"
    - key: "debug_list"  # send to debug_list if `message` field contains DEBUG
      when.contains:
        message: "DEBUG"
    - key: "%{[fields.list]}"

Copy

explain ： default key The value is default_list,keys The value of is created by dynamic allocation , When redis In the received log message The value of the field contains error Field , Create key by error_list, When the package contains DEBUG Field , Create key by debug_list.

  The solution to the problem is to add a value that can distinguish the log in the output log of each application , And then in keys Set in , In this way, the logs of different applications can be output to different redis Of key in .