Project practice 5: build elk log collection system

This paper mainly summarizes the overall idea and kibana Custom charts and Dashboard

The whole idea is as follows ：

Train of thought ： Our program or app After log generation , from Filebeat To collect , Send to Logstash,Logstash Perform segmentation processing, etc , Send after processing  es in , Finally by Kibana To display ,Kibana When displaying, you can customize charts, etc

One ：Elasticsearch And Kibana install

        Reference address ：Elasticsearch And Kibana install _cc_ Nanke Yimeng's blog -CSDN Blog

Two ：Logstash Installation and use

        Reference address ：Logstash Installation and use _cc_ Nanke Yimeng's blog -CSDN Blog

        The configuration in the reference article is a direct read file , The configuration file needs to be modified

        ​​​​​​​ 

        Screenshot of original log

        

        To configure filter explain ： use | Segmentation , After splitting, an array will be generated ,  Add a field to the log and send it to es

        

input {
  beats{
    port => "5044"
  }
}

filter {
  mutate {
    split => {"message"=>"|"}
  }
  mutate {
    add_field => {
      "cc" => "%{[message][3]}"
      "visit" => "%{[message][4]}"
    }
  }
  mutate {
    convert => {
      "cc" => "string"
      "visit" => "string"
    }
  }

}

output {
  elasticsearch {
    hosts => ["192.168.231.147:9200"]
   }
}

3、 ... and ：Filebeat Installation and use

        Reference address ：Filebeat Installation and use _cc_ Nanke Yimeng's blog -CSDN Blog

        Pay attention to the reference article filebeat It's sent directly to es in , At this point, we need to modify the configuration file , Send the collected logs to Logstash, The configuration is as follows

        

Four ：kibana Custom charts and Dashboard

        【1】 establish pattern

        

        

        

        【2】 Create column chart

                1、 Start to create

                 

                 

                

                 2、 You can set a timeline         ​​​​​​​    

                

                 You can set the time period for reading data , Statistical interval , Refresh time, etc

                 

 

                 3、 Preservation

                

        【3】 Create a pie chart distribution

                

                 

                 Add fields , Set properties , Generate the pie chart you want

                 

                 

                  Preservation

                

         【4】 Data table

                add The data we want

                

                  Preservation

                

        【5】 Make Dashboard

                

                 

                Click to

                 

                You can drag the location , And size . Adjust to the style you want

        ​​​​​​​        

                  Also save

                

 

So our elk The log management system has been set up .