Read the logstash principle

One .Logstash

Logstash As Elasicsearch Common real-time data acquisition engine , You can collect data from different data sources , The data is processed and output to a variety of output sources , yes Elastic Stack  An important part of .

Two .Logstash Hello world

cd logstash-6.4.0
bin/logstash -e 'input { stdin { } } output { stdout {} }' 


Logstash Standard input and standard output will be used as input and output, And don't specify filter. among ,syslog  For the system log . Save the file . On the console, the results are as follows ：

3、 ... and .action

index  Index a document delete  adopt id Value to delete a document （ This action One needs to be specified id value ）create  Insert a document message , If this document information already exists in the index , So this insertion failed update  adopt id Value to update a document . Update has a special case upsert, If the updated document doesn't exist yet , Then it will use upsert

action => "index"


default Logstash The installation contains Beats input plug-in unit ,Beats input Plugins allow Logstash from Elastic Beats The framework receives events , That is to say, any Beat written Use Beats Framework work , for example Packetbeat and Metricbeat, You can also send event data to Logstash.

Four .manage_template

Boolean type   The default is true Set to false Will close logstash Automatic template management functions, such as you define a custom template , More field names dynamically generate fields , Then it should be set to false

5、 ... and .filter

Logstash The second component of the three components , It's true Logstash The most complex tool , One of the most painful components , Of course , It is also the most useful component .

filter{
 grok{
 match => ['message','%{TIMESTAMP_ISO8601:logdate}']
 }
}


date plug-in unit   Here we need to close the front grok The value stripped by the plug-in logdate In combination with

filter{
 date{
 match=>["logdate","dd/MMM/yyyy:HH:mm:ss Z"]
 target=>"@timestamp"
 remove_field => 'logdate'
 }
}