当前位置：网站首页>Elk distributed log analysis system deployment (Huawei cloud)

Elk distributed log analysis system deployment (Huawei cloud)

2022-07-05 19:52:00 【51CTO】

Architecture diagram

ELK Deployment of distributed log analysis system （ Hua Wei Yun ）_ Log system

One 、 Environmental preparation

1. Host list

Host name	IP Address	To configure
es-0001	192.168.1.41	1cpu,1G Memory ,10G Hard disk
es-0002	192.168.1.42	1cpu,1G Memory ,10G Hard disk
es-0003	192.168.1.43	1cpu,1G Memory ,10G Hard disk
es-0004	192.168.1.44	1cpu,1G Memory ,10G Hard disk
es-0005	192.168.1.45	1cpu,1G Memory ,10G Hard disk
Kibana	192.168.1.46	1cpu,1G Memory ,10G Hard disk
logstash	192.168.1.47	2CPU、2G Memory ,10G Hard disk
apache	192.168.1.48	1PU、1G Memory ,10G Hard disk

2. Software inventory

elasticsearch-2.3.4.rpm
logstash-2.3.4-1.noarch.rpm
kibana-4.5.2-1.x86_64.rpm
filebeat-1.2.3-x86_64.rpm

3. Plug in list ：

[x] elasticsearch plug-in unit ：

software package	describe

| bigdesk-master.zip | yes ES Cluster monitoring tools |

| elasticsearch-kopf-master.zip | One ElasticSearch Management tools , Provide right ES Cluster operated API |

| elasticsearch-head-master.zip | show ES Clustered # Display help information ?help topology , Indexable （index)、 node (node) Level of operation |

4. Huawei cloud jump board

To configure yum repositories ：

       
       [[email protected] <sub>]
       
       # rm -rf /etc/yum.repos.d/*.repo
       
[[email protected] </sub>]
       
       # curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.myhuaweicloud.com/repo/CentOS-Base-7.repo
       
[[email protected] <sub>]
       
       # yum clean all
       
[[email protected] </sub>]
       
       # yum makecache
       
[[email protected] <sub>]
       
       # yum install -y net-tools lftp rsync psmisc vim-enhanced tree vsftpd bash-completion createrepo lrzsz iproute
       
[[email protected] </sub>]
       
       # mkdir /var/ftp/localrepo
       
[[email protected] <sub>]
       
       # cd /var/ftp/localrepo
       
[[email protected] </sub>]
       
       # createrepo .
       
[[email protected] <sub>]
       
       # createrepo --update . #  to update 
       
[[email protected] </sub>]
       
       # systemctl enable --now vsftpd
       
[[email protected] ~]
       
       # cp -a elk /var/ftp/localrepo/elk
       
[[email protected] elk]
       
       # cd /var/ftp/localrepo/
       
[[email protected] localrepo]
       
       # createrepo --update .
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

Optimize system services

       
       [[email protected] <sub>]
       
       # systemctl stop postfix
       
[[email protected] </sub>]
       
       # yum remove -y postfix firewalld-*
       
[[email protected] <sub>]
       
       # yum install chrony
       
[[email protected] </sub>]
       
       # vim /etc/chrony.conf 
       
       #  Comment out  server  Start line , Add the following configuration 
       
server ntp.myhuaweicloud.com minpoll 
       
       4 maxpoll 
       
       10 iburst
       
[[email protected] <sub>]
       
       # systemctl enable --now chronyd
       
[[email protected] </sub>]
       
       # chronyc sources -v // Verify configuration results  ^*  On behalf of success 
       
[[email protected] <sub>]
       
       # vim /etc/cloud/cloud.cfg
       
       # manage_etc_hosts: localhost // Comment out this line 
       
[[email protected] </sub>]
       
       # reboot
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Installation configuration ansible Management host

       
       [[email protected] <sub>]
       
       # cp -a ansible_centos7 /var/ftp/localrepo/ansible
       
[[email protected] </sub>]
       
       # cd /var/ftp/localrepo
       
[[email protected] <sub>]
       
       # createrepo --update .
       
[[email protected] </sub>]
       
       # vim /etc/yum.repos.d/local.repo 
       
  [local_repo]
       
       name
       
       =CentOS-
       
       $releasever – Localrepo
       
       baseurl
       
       =ftp://192.168.1.252/localrepo
       
       enabled
       
       =
       
       1
       
       gpgcheck
       
       =
       
       0
       
[[email protected] <sub>]
       
       # yum makecache
       
[[email protected] </sub>]
       
       # yum install -y ansible // Go to Huawei cloud website to download the secret key , And upload the secret key to the springboard 
       
[[email protected] <sub>]
       
       # mv luck.pem /root/.ssh/id_rsa
       
[[email protected] </sub>]
       
       # chmod 0400 /root/.ssh/id_rsa
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

5. Huawei cloud template machine （ Mirror image ） To configure

       
       [[email protected] <sub>]
       
       # passwd root
       
[[email protected] </sub>]
       
       # rm -rf /etc/yum.repos.d/*.repo
       
[[email protected] <sub>]
       
       # curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.myhuaweicloud.com/repo/CentOS-Base-7.repo
       
[[email protected] </sub>]
       
       # vim /etc/yum.repos.d/local.repo 
       
[local_repo]
       
       name
       
       =CentOS-
       
       $releasever – Localrepo
       
       baseurl
       
       =ftp://192.168.1.252/localrepo
       
       enabled
       
       =
       
       1
       
       gpgcheck
       
       =
       
       0
       
[[email protected] <sub>]
       
       # yum clean all
       
[[email protected] </sub>]
       
       # yum makecache
       
[[email protected] <sub>]
       
       # yum repolist
       
[[email protected] </sub>]
       
       # yum install -y net-tools lftp rsync psmisc vim-enhanced tree lrzsz bash-completion iproute
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

Optimize system services

       
       [[email protected] <sub>]
       
       # systemctl stop postfix atd tuned
       
[[email protected] </sub>]
       
       # yum remove -y postfix at audit tuned kexec-tools firewalld-*
       
[[email protected] <sub>]
       
       # yum install chrony
       
[[email protected] </sub>]
       
       # vim /etc/chrony.conf 
       
       #  Comment out  server  Start line , Add the following configuration 
       
 server ntp.myhuaweicloud.com minpoll 
       
       4 maxpoll 
       
       10 iburst
       
[[email protected] <sub>]
       
       # systemctl enable --now chronyd
       
[[email protected] </sub>]
       
       # chronyc sources -v
       
       #  Verify configuration results  ^*  On behalf of success 
       
[[email protected] <sub>]
       
       # vim /etc/cloud/cloud.cfg
       
       # manage_etc_hosts: localhost  Comment out this line 
       
[[email protected] </sub>]
       
       # yum clean all 
       
[[email protected] ~]
       
       # poweroff
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

1. Host name resolution （ mutual ping through ）

       
       # vim /etc/hosts // All host configurations 
       
       192.168.1.41    es-0001
       
       192.168.1.42    es-0002
       
       192.168.1.43    es-0003
       
       192.168.1.44    es-0004
       
       192.168.1.45    es-0005
       
       192.168.1.46    kibana
       
       192.168.1.47    logstash
       
       192.168.1.48    apache
      
1.
2.
3.
4.
5.
6.
7.
8.
9.

Two 、【 Deploy Elasticsearch】( all es node )

A. Deploy elasticsearch service

Install the software

       
       # yum install -y java-1.8.0-openjdk elasticsearch // edition ：2.3.4.rpm
       
       # java -version
       
  openjdk version 
       
       "1.8.0_161"
       
  OpenJDK Runtime Environment (build 
       
       1.8.0_161-b14)
       
  OpenJDK 
       
       64
       
       -Bit Server VM (build 
       
       25.161-b14, mixed mode)
       
       # sestatus // see selinux state 
      
1.
2.
3.
4.
5.
6.

Modify the configuration

       
       # vim /etc/elasticsearch/elasticsearch.yml
       
       17| cluster.name: my-ES        // Configuration cluster name 
       
       23| 
       
       node.name: es1        // Current host name 
       
       55| network.host:0.0.0.0      // Listen to all addresses / This machine IP
       
       68| discovery.zen.ping.unicast.hosts:  [
       
       "es1",
       
       "es2",
       
       "es3"]    // Declare cluster members （ Not all ）
       
 perhaps 
       
       # sed -i '/cluster.name/s/# cluster.name/cluster.name/' /etc/elasticsearch/elasticsearch.yml // Remove annotations 
       
       # sed -i '/cluster.name/s/my-application/my-ES/' /etc/elasticsearch/elasticsearch.yml 
       
       # sed -i '/node.name/s/# node.name: node-1/node.name: es-0002/' /etc/elasticsearch/elasticsearch.yml
       
       # sed -i '/network.host/s/# network.host: 192.168.0.1/network.host: 0.0.0.0/' /etc/elasticsearch/elasticsearch.yml
       
       # sed -i '/discovery.zen.ping/a discovery.zen.ping.unicast.hosts: ["es-0001", "es-0002"]' /etc/elasticsearch/elasticsearch.yml // Add configuration under the line 
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

Start the service

      
      
       
       # systemctl start elasticsearch.service 
       
       

       
       # systemctl enable elasticsearch.service
      
      
      
      
       
       1.
       
       2.

Service validation

       
       # ss -ntulp | grep 9200
       
       # ss -ntulp | grep 9300
       
       # curl http://es-0001:9200/
       
       # curl http://192.168.1.41:9200/_cluster/health?pretty 
       
{
       
       "cluster_name" : 
       
       "my-ES",     // Cluster identity / name 
       
       "status" : 
       
       "green",
       
       "timed_out" : 
       
       false,
       
       "number_of_nodes" : 
       
       3,      // Number of cluster members 
       
       "number_of_data_nodes" : 
       
       3,   // How many nodes are there in the cluster   
       
       "active_primary_shards" : 
       
       0,
       
       "active_shards" : 
       
       0,
       
       "relocating_shards" : 
       
       0,
       
       "initializing_shards" : 
       
       0,
       
       "unassigned_shards" : 
       
       0,
       
       "delayed_unassigned_shards" : 
       
       0,
       
       "number_of_pending_tasks" : 
       
       0,
       
       "number_of_in_flight_fetch" : 
       
       0,
       
       "task_max_waiting_in_queue_millis" : 
       
       0,
       
       "active_shards_percent_as_number" : 
       
       100.0
       
}
       
cluster： colony 
       
health： health 
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.

B. Plug in deployment language application

Which machine is the plug-in installed on , Only use on which machine

       
        For local installation file; For remote installation ftp.
       
       # /usr/share/elasticsearch/bin/plugin install file:///root/file/bigdesk-master.zip
       
       # /usr/share/elasticsearch/bin/plugin install file:///root/file/elasticsearch-kopf-master.zip
       
       # /usr/share/elasticsearch/bin/plugin install file:///root/file/elasticsearch-head-master.zip
       
       # /usr/share/elasticsearch/bin/plugin list // Check the installed plug-ins 
       
Installed plugins 
       
       in /usr/share/elasticsearch/plugins:
       
       - kopf
       
       - bigdesk
       
       - head
      
1.
2.
3.
4.
5.
6.
7.
8.
9.

Access plug-ins

1、 Huawei cloud is bound to the elastic public network IP to es-0005 node
2、http:// Elastic public network IP:9200/_plugin/ The plug-in name [bigdesk|head|kopf]
http:// Public network IP:9200/\_plugin/kopf
http:// Public network IP:9200/\_plugin/head
http:// Public network IP:9200/\_plugin/bigdesk
http:// Public network IP/info.php // Request method

Elasticsearch Basic operation

Inquire about _cat Method

       
       [[email protected] <sub>]
       
       # curl -XGET http://es-0001:9200/_cat/ #  Query supported keywords 
       
[[email protected] </sub>]
       
       # curl -XGET http://es-0001:9200/_cat/master #  Check the specific information 
       
[[email protected] <sub>]
       
       # curl -XGET http://es-0001:9200/_cat/master?v #  Show details  ?v
       
[[email protected] </sub>]
       
       # curl -XGET http://es-0001:9200/_cat/master?help #  Display help information  ?help
       
1.
2.
3.
4.

Create index

Specify the name of the index , Specify the number of tiles , Specify the number of copies

Create an index using PUT Method , After the creation is completed, pass head Plug in validation

       
       [[email protected] ~]
       
       # curl -XPUT http://es-0001:9200/tedu -d \
       
       '{
       
        "settings":{ // establish 
       
        "index":{ // Indexes 
       
        "number_of_shards": 5, // Number of slices 
       
        "number_of_replicas": 1 // Copy number 
       
        }
       
        }
       
       }'
       
 It is equivalent to building a database 
       
       # curl -XGET http://es-0001:9200/_cat/indices?v // View index details 
       
 Browser access ：http://122.9.96.62:9200/_plugin/head/     The change of , The block depth is the source data , Shallow copy 
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

![image-20210512185824453](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512185824453.png)

Add data

       
       # curl -XPUT http://es-0001:9200/tedu/teacher/1 -d \ / Indexes ( library ）/ type （ Table name ）/id
       
       '{
       
        " occupation ": " Poet ",
       
        " name ": " Li Bai ",
       
        " The title of ": " Poetry fairy ",
       
        " years ": " The tang dynasty "
       
       }' 
      
1.
2.
3.
4.
5.
6.
7.

Query data

      
      
       
       [[email protected] ~]
       
       # curl -XGET http://es-0001:9200/tedu/teacher/1?pretty
      
      
      
      
       
       1.

Modifying data

       
       [[email protected] ~]
       
       # curl -XPOST http://es-0001:9200/tedu/teacher/1/_update -d \
       
       '{ 
       
        "doc": {
       
        " years ": " A.D. 701"
       
        }
       
       }'
      
1.
2.
3.
4.
5.
6.

Delete data

      
      
       
       [[email protected] <sub>]
       
       # curl -XDELETE http://es-0001:9200/tedu/teacher/1 // Delete one 
       
       
[[email protected] </sub>]
       
       # curl -XDELETE http://es-0001:9200/tedu // Delete index 
      
      
      
      
       
       1.
       
       2.

3、 ... and .Kibana Deploy

Upload kibana-4.5.2-1.x86_64.rpm

       
       # yum install -y kibana
       
       # vim /opt/kibana/config/kibana.yml // Modify the configuration file 
       
       2| server.port: 
       
       5601                // port 
       
       5| server.host: 
       
       "0.0.0.0"
       
       15| elasticsearch.url: http://es1:9200          // colony 
       
       23| kibana.index: 
       
       ".kibana"
       
       26| kibana.defaultAppId: 
       
       "discover"        //kibana default page 
      
1.
2.
3.
4.
5.
6.
7.

Service startup / verification

       
       # systemctl start kibana
       
       # systemctl enable kibana
       
       # ss -ntulp | grep 5601
      
1.
2.
3.

Access test

http:// Elastic public network IP:5601/status

Import log data

Copy cloud disk public/elk/logs.jsonl.gz To the springboard machine

      
      
       
       [[email protected] ~]
       
       # gunzip logs.jsonl.gz 
       
       

       
       # curl -XPOST http://192.168.1.41:9200/_bulk --data-binary @logs.jsonl // It will take a few minutes 
      
      
      
      
       
       1.
       
       2.

Successful import ：

![image-20210512214744461](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512214744461.png)

To configure kibana, wildcard

![image-20210512215605715](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512215605715.png)

The modification time is consistent with the log

![image-20210512215914935](C:\Users\ Uncle gou \AppData\Roaming\Typora\typora-user-images\image-20210512215914935.png)

【logstash】

install logstash

       
       # yum install -y java-1.8.0-openjdk logstash
       
       # vim /etc/logstash/logstash.conf // Create the configuration file manually 
       
input { 
       
  stdin {}
       
}
       
filter{ }
       
output{ 
       
  stdout{}
       
}
       
stdin   The standard input （0）
       
stdout   standard output （1）
       
stderr   Error output （2） 
       
       # /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf // Start service from profile 
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

Plug in and debugging format

Use json Format string test {"a":"1", "b":"2","c":"3"}

       
       # vim /etc/logstash/logstash.conf
       
input { 
       
  stdin { codec 
       
       => 
       
       "json" }
       
}
       
filter{ }
       
output{ 
       
  stdout{ codec 
       
       => 
       
       "rubydebug" }
       
}
       
       # /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
      
1.
2.
3.
4.
5.
6.
7.
8.
9.