Filebeat custom indexes and fields

Like logstash, By default filebeat-* The index of and use the... Under the unzipped directory fields.yml( Binary compression package )

The requirement here is to parse the generated specific file , Each line is recorded as json Formatted data , need ：

• Different files generate different indexes ;
• Type needs to be defined , The string was not used in the original query .keyword, Because the default template uses less than 1024 yes keyword, Conversely text, If you want to be keyword Use is fieldName.keyword;

To configure

setup.template.name: custom_name
setup.template.pattern: custom_name_*
setup.template.enabled: false
setup.template.overwrite: false
setup.template.fields: customFields.yml
setup.ilm.enabled: false
processors:
- drop_fields:
    fields: [log, host, input, agent, ecs]
    ignore_missing: false
filebeat.inputs:
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t1]
  clean_removed: true
  enabled: true
  json: {
    keys_under_root: true, overwrite_keys: true}
  paths: [/data/t1_*]
- close_removed: true
  close_inactive: 5m
  type: log
  tags: [t2]
  clean_removed: true
  enabled: true
  json: {
    keys_under_root: true, overwrite_keys: true}
  paths: [/data/t2_*]
output.elasticsearch:
  indices:
  - index: custom_name_t1
    when.contains: {
    tags: t1}
  - when.contains: {
    tags: t2}
    index: custom_name_t2
  hosts: ['127.0.0.1:9200']

setup.template.name Set up a new template , The name of the template
setup.template.pattern The template matches those indexes
setup.template.enabled: false Turn off the default template configuration
setup.template.overwrite: false Whether to overwrite the existing template
when.contains: contain
keys_under_root: true take field Expand to outermost (the custom fields are stored as top-level fields in the output document)

Please view the customized template ：Configuration-template

Please check the file path ：Configure project paths

Field definition

setup.template.fields: customFields.yml( Please see the above link for the path ), The file format is as follows ：

- key: custom_name
  title: custom_name
  description: > custom fields
  fields:
  # some desc
    - name: t1
      type: keyword
    - name: t2
      type: keyword
    - name: t3
      type: ip
    - name: t4
      type: integer
    - name: t5
      type: ip
    - name: t6
      type: integer
    - name: t7
      type: ip
    - name: t8
      type: text
    - name: t9
      type: date
    - name: t10
      type: long