Application system log structure of elastic stack

List of articles

Elastic Stack Log collection of Logstash
Elastic Stock Log collection of ： Optimization

---

---

Preface

Logstash The log collection process is roughly divided into ： Input 、 Filters and outputs ; Three steps .

My architecture plan is ：
logback adopt rabbitmq Pass the log to Logstash Real time filtering and structuring through plug-ins ,
Another solution is to use in the application system logstash-logback-encoder To transcode logs , The effect is like https://blog.csdn.net/wangxudongx/article/details/103743963.

Here we introduce the scheme of filtering through plug-ins .

Logstash It can collect data dynamically 、 Converting and transferring data , Not affected by format or complexity . utilize Grok Derive structure from unstructured data , from IP The address decodes the geographic coordinates , Anonymous or exclude sensitive fields , And simplify the whole process .

Screening
Real time analysis and conversion of data
The process of transferring data from the source to the repository ,Logstash Filters can parse Events , Identify the named fields to build the structure , And convert them to a common format , For more powerful analysis and business value .

Logstash Be able to transform and parse data dynamically , Not affected by format or complexity ：

utilize Grok Derive structure from unstructured data
from IP The address deciphers the geographic coordinates
take PII data anonymization , Completely exclude sensitive fields
Simplify the overall treatment , Not affected by data source 、 The impact of format or architecture
Use our rich filter library and versatile Elastic Common Schema, You can realize infinite possibilities .

You want to convert the application system log into structured data and put it into ES We need to do something , There are currently two schemes ; One is to use on the application side logstash-logback-encoder library , The other is to use Logstash Provided filter Plug in to do , Here we use filter plug-in unit , of logstash-logback-encoder Please see the same series of articles of bloggers .

---

Tips ： The following is the main body of this article , The following cases can be used for reference

Environmental Science

Components	edition
CentOS	7
Docker	20.10.7
elasticsearch	7.6.2
logstash	7.6.2
kibana	7.6.2
rabbitmq	3.8.9-management

Logstash filter（ Screening ）

Real time analysis and conversion of data

The process of transferring data from the source to the repository ,Logstash Filters can parse Events , Identify the named fields to build the structure , And convert them to a common format , For more powerful analysis and business value .

Logstash Be able to transform and parse data dynamically , Not affected by format or complexity ： utilize Grok Derive structure from unstructured data from IP The address deciphers the geographic coordinates take PII data anonymization , Completely exclude sensitive fields Simplify the overall treatment , Not affected by data source 、 The impact of format or architecture Use a rich filter library and a variety of functions Elastic Common Schema, You can realize infinite possibilities .

Grok Filter plug in

Grok yes Logstash One of many filter plug-ins , It is a good way to parse unstructured log data into structured and queriable things . The principle is to use regular expressions .

Here we use the plug-in version ：v4.2.0

By default ,Logstash With about 120 Patterns . You can find them here ：https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
You can simply add your own .（ see patterns_dir Set up ）

Grok still Dissect？ Or both ？
The dissect Filter plug-ins are another way , To extract unstructured event data to use delimiter fields .

Dissect And Grok The difference is that it does not use regular expressions and is faster . When data repeats reliably , The anatomical effect is very good . When your text structure varies from line to line ,Grok It's a better choice .

You can use Dissect and Grok For mixed use cases , When a part of the line repeats reliably , But the whole line is not .Dissect Filters can deconstruct repeated line segments .Grok Filters can handle residual field values with more regular expression predictability .

I'm not right here DIssect Introduced , If you are interested, you can read the document by yourself .

grok Official documents
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

Grok Basic knowledge of

Grok The regular expression library used is Oniguruma

Grok It works by combining text patterns into content that matches your log .

grok The syntax of the pattern is %{SYNTAX:SEMANTIC:TYPE}

SYNTAX yes , Will match the name of the text pattern . Frankly speaking, it's a period Logstash Or the name of your custom regular expression .

for instance MINUTE (?:[0-5][0-9])
MINUTE Namely (?:[0-5][0-9])` The name of this expression , Can be in pipeline Of grok Segment usage .

The SEMANTIC It is the field name generated after you match the identifier of a paragraph of text . for example ,35 It may be the duration of the event , So you can simply call it minute.

For the example above , Your grok The filter will be as follows ：

%{
     MINUTE: The duration of the }

You can also choose to send your grok Schema add data type conversion . By default , All semantics are saved as strings . If you want to convert semantic data types , For example, changing a string to an integer , Then use the target data type as the suffix . for example %{NUMBER:num:int}, take num Semantic conversion from string to integer . The only conversion supported at the moment is int and float.

Self contained grok Method

cat /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns

https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

To configure pipeline

pipeline To configure

If in Logstash No suitable matching pattern is found in the matching pattern provided by default , We can also use regular expressions to customize patterns .
Such as the following configuration items patterns_dir Is the file that specifies the custom configuration mode .

Example of configuration file content ：

USERNAME [a-zA-Z0-9._-]+
USER %{
    USERNAME}

input {
    
  rabbitmq {
    
                host => "3.8.23.16"
                port => 5672
                user => admin
                password => "blke134343"
                durable => true
                queue => "q_logstash"
                codec => plain                                                                        
    }
}

 filter{
    
       grok{
    
                patterns_dir => "/usr/share/logstash/my-grok-pattern"
               match => {
     "message" => "%{DATESTAMP:logTime} %{THREAD:thread} %{LOGLEVEL:logLevel} %{GREEDYDATA:logContent}"
                        }
                remove_field => ["message"]
       }

}

output {
    
        stdout {
     codec => rubydebug  }
            elasticsearch {
     hosts => "172.17.0.3:9200"  }       
}

Original log

2021-07-20 10:26:24.952 [http-nio-6002-exec-9] DEBUG c.rocky.chats.mapper.ChatsUserinfoMapper.selectOne.? ? - ==>  Preparing: SELECT create_by,create_time,update_by,update_time,id,nickname,sex,short_id,level,mobile,email,channel,deleted,headimg,pwd,birthday FROM chats_userinfo WHERE email = ? AND pwd = ? \n

{
    
      "@version" => "1",
    "@timestamp" => 2021-07-20T10:26:24.994Z,
       "message" => "2021-07-20 10:26:24.952 [http-nio-6002-exec-9] DEBUG c.rocky.chats.mapper.ChatsUserinfoMapper.selectOne.? ? - ==> Preparing: SELECT create_by,create_time,update_by,update_time,id,nickname,sex,short_id,level,mobile,email,channel,deleted,headimg,pwd,birthday FROM chats_userinfo WHERE email = ? AND pwd = ? \n"
}

grok Log after conversion

{
    
      "@version" => "1",
    "@timestamp" => 2021-07-20T10:24:33.307Z,
       "logTime" => "21-07-20 10:24:33.251",
    "logContent" => "c.rocky.chats.mapper.ChatsUserinfoMapper.selectOne.? ? - ==> Preparing: SELECT create_by,create_time,update_by,update_time,id,nickname,sex,short_id,level,mobile,email,channel,deleted,headimg,pwd,birthday FROM chats_userinfo WHERE email = ? AND pwd = ? \n",
        "thread" => "[http-nio-6002-exec-4]",
      "logLevel" => "DEBUG"
}

stay Kibana Inside debug grok sentence

If you need to customize the matching pattern or construct a general Grok sentence , So grammar debugger Tools will be very useful to you .

menu ：Management -> Dev Tools -> Grok Debugger

Another debugging Grok Pattern The place of
https://grokdebug.herokuapp.com/

I don't think so Kibana good-looking .

stay Kibana It manages application system logs

You need to create Kibana Of Index Patterns

stay Discover Read the log inside

Create a chart

Save the query

Create a dashboard

---

summary

At present, the flat log is basically enough for development 、 The operation and maintenance personnel used , To find more useful information through logs, we need to store logs as data and analyze them with other tools , And warehousing must be structured , So this is the application scenario of log structure .

Of course, if you want to make the log more useful, you also need developers to make the log output as brief as possible , Form planned log output , This is more conducive to the subsequent use of logs .