Establishment of elk log analysis system

ELK brief introduction

What is a journal
Logs are generated by programs , Follow a certain format （ Usually contains a timestamp ） Text data .
 

ELK Common architectures

Elasticsearch + Logstash + Kibana
This is the simplest Architecture . This architecture , adopt logstash Collect the logs ,Elasticsearch Analysis log , And then in Kibana(web Interface ) Show in . Although this kind of architecture is introduced on the official website
Usually logs are generated by the server , Output to different files , There are usually system logs 、 Application log 、 Security log . These logs are stored on different machines .
 

Based on the environment ：

operating system ：Centos  Linux release 7.4.1708

Turn off firewall 、selinux

sed -ri '/^[^#]*SELINUX=/s#=.+$#=disabled#' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld

Host name resolution ：192.168.100.10      ELK

Elasticsearch Deploy

add to yum Warehouse

 add to yum Warehouse 
vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Download and install elasticsearch Of yum Source's key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

install elasticsearch

yum install -y elasticsearch

install java Environmental Science

 yum install -y  java
[[email protected] /]# java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)

Modify the configuration file

vim /etc/elasticsearch/elasticsearch.yml  

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 0.0.0.0

http.port: 9200

xpack.security.enabled: false

  Create folder And authorize As above pat.data=

[[email protected] ]# mkdir  /data/elasticsearch 
[[email protected]]#  chown -R elasticsearch:elasticsearch  /data/elasticsearch/

start-up

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service

start-up web test

ss -anpt |egrep "9200|9300" 
LISTEN     0      128         :::9200                    :::*                   users:(("java",pid=2053,fd=372))
LISTEN     0      128         :::9300                    :::*                   users:(("java",pid=2053,fd=368))
ESTAB      0      0         ::ffff:127.0.0.1:9200                  ::ffff:127.0.0.1:49888               users:(("java",pid=2053,fd=544))
ESTAB      0      0         ::ffff:127.0.0.1:9200                  ::ffff:127.0.0.1:49698               users:(("java",pid=2053,fd=438))

web test ：http://192.168.100.10:9200

  Download and install logstash  start-up logstash

yum install -y logstash
systemctl restart elasticsearch

  Modify the configuration file

vim /etc/logstash/logstash-sample.conf
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

 tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
   }

   rabbitmq {
        host=>"localhost"
        vhost => "/"
        port=> 5672
        user=>"guest"
        password=>"guest"
        queue=>"station_Route"
        durable=> true
        codec=>json
 }

output {
  elasticsearch {
    hosts => ["http://ip:9200"]
    index => "rabbitmq-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

install kibana

yum install kibana.x86_64 -y

  Modify the configuration file

vim /etc/kibana/kibana.yml 
 
6 That's ok    server.port: 5601 
11 That's ok   server.host: "0.0.0.0"
32 That's ok   server.name: "test-kin"
43 That's ok   elasticsearch.hosts: ["http://localhost:9200"]
114 That's ok  i18n.locale: "zh-CN"

start-up kibana

systemctl start kibana.service
systemctl restart kibana.service

The visit address is ：http://192.168.100.10:5601/

  Click Browse by yourself