Elk project monitoring platform deployment + deployment of detailed use (II)

Because there may be some differences between different version numbers , So I'm here to ensure the smooth implementation of the next step . First confirm the version number and the deployment environment .

elasticsearch 5.5.1

logstash 5.5.1

kibana  5.5.1

These three servers are deployed in one server . Those with large business volume can consider elasticsearch Separate and cluster .

For convenience, the following are in es To represent the elasticsearc.

es + kibana Choose to use it for the convenience of being unfamiliar docker

stay linux Lower installation docker After execution ：

docker pull docker.elastic.co/elasticsearch/elasticsearch:5.5.1

docker pull docker.elastic.co/kibana/kibana:5.5.1

logstash Use the source code to install ：

wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.1.tar.gz

cp logstash-5.6.1.tar.gz /usr/share/logstash.tar.gz

tar -zxvf  logstash.tar.gz

cd  logstash

./bin/logstash -e 'input { stdin {} } output { stdout {} }'

Test the installation for success

Now try starting es and kibana：

elasticsearch start-up ：

docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" --name my-elastic -d docker.elastic.co/elasticsearch/elasticsearch:5.5.1

kibana start-up ：

docker run -p 5601:5601 -e "ELASTICSEARCH_URL=http://localhost:9200" --name my-kibana --network host -d docker.elastic.co/kibana/kibana:5.5.1

At this time, if everything is implemented smoothly, you should be able to access kibana 了

The access address is deployed for you http://ip:5601 visit

Of course, there is no data at this time , Now on the project server Deploy data collection filebeat and metricbeat

1. Download these two files in the project server ：

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-linux-x86_64.tar.gz

wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.6.1-linux-x86_64.tar.gz

2. decompression

tar -zxvf *.tar.gz

3. To configure filebeat With ngxin For example, log

cd filebeat-5.6.1-linux-x86_64

vim nginx.yml

Copy the contents of the following file

filebeat.prospectors:

- input_type: log

  paths:

    - /var/log/nginx/access.log

  document_type: nginx_access

  # Then group and count the logs

  fields:

      level: debug

  #Filebeat Start from the end of the file to monitor what's new in the file , Send each new line of file as an event in turn

  tail_files: true

shipper:

    tags: ['nginx-access']

# Project name and project service ip The address can be customized

tags: ["myserver", "101.110.56.78"]

output.logstash:

   # logstash Server's ip Address

   hosts: ["11.142.42.77:5044"]

---

shipper yes logstash Condition control is used when receiving logs

output.logstash To configure logstash Server's ip Address + Port number

start-up filebeat

./filebeat -e -c ./nginx.yml -d "publish"

To configure logstash

Get into logstash The installation directory

vim logstash.yml

input {

beats {

port => 5044

}

}

filter {

        if [type]  == "nginx_access"  {

          ruby {

                init => "@kname = ['remote_addr','remote_user','time_local','request','status','body_bytes_sent','http_referer','http_user_agent','http_x_forwarded_for']"

                code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('message').split(' | '))]))"

          }

          if [request] {

               ruby {

                    init => "@kname = ['method','uri','verb']"

                    code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))]))"

          }

          if [uri] {

                ruby {

                    init => "@kname = ['url_path','url_args']"

                    code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('request').split('?'))]))"

                }

                kv {

                    prefix => "url_"

                    source => "url_args"

                    field_split => "& "

                    remove_field => [ "url_args","uri","request" ]

                }

          }

          }

          mutate {

                convert => [ "body_bytes_sent" , "integer" ]

          }

          date {

                match => [ "time_local", "dd/MMM/yyyy:hh:mm:ss Z" ]

                locale => "en"

          }

        }

}

output {

      if [type] == "nginx_access" {

        stdout {

            codec => rubydebug

        }

        elasticsearch {

                      hosts => ["localhost:9200"]

                      index => "nginx_access_%{+YYYY.MM.dd}"

                      user => "elastic"

                      password => "changeme"

        }

        if [status] != "200"{

                exec {

                    command  =>  "sh /root/sh/alarm.sh %{tags[0]} Found out BUG, Please deal with as soon as possible The server IP:%{tags[1]} request:%{request} Access status :%{status} Time :%{time_local}"

        }

        }

      }

      else {

        stdout {

            codec => rubydebug

        }

        elasticsearch {

              hosts => ["localhost:9200"]

              index => "metricbeat-%{+YYYY.MM.dd}"

              user => "elastic"

              password => "changeme"

        }

      }

}

---

It should be noted that ：

1.nginx.conf You need to configure nginx The format of is convenient for logstash analytical

log_format main "$remote_addr | $remote_user | $time_local | $request | $status | $body_bytes_sent | $http_referer | $http_user_agent | $http_x_forwarded_for";

access_log /var/log/nginx/access.log main;

2. Need to install logstash plug-in unit exec To execute the script

3. Script alarm.sh The content is ：

start-up logstash：

./logstash -f ./logstash.yml