ELK Enterprise log analysis system

one 、ELK brief introduction

• ELK The platform is a complete set of centralized log processing solutions , take ElasticSearch、Logstash and Kiabana Three open source tools are used together , Complete more powerful user query of logs 、 Sort 、 Statistical needs .

ElasticSearch：

ElasticSearch Is based on Lucene ( The architecture of a full-text search engine ） Distributed development Storage retrieval engine , Used to store all kinds of logs .

ElasticSearch Yes, it is Java Developed , It can be done by RESTful Web Interface , So that users can communicate with ElasticSearch signal communication .

ElasticSearch It's a real-time 、 Distributed and scalable Search engine , Allow full text 、 Structured search

ElasticSearch Usually Used to index and search large volumes of log data , It can also be used to search for many different types of documents .

Kiabana：

Kibana Usually with ElasticSearch Deploy together ,Kibana yes BlasticSearch A powerful data visualization Dashboard,

Kibana Provide graphical Web Interface to browse ElasticSearch Log data , Sure Used to summarize 、 Analyze and search important data .

Logstash：

Logstash As a log data collection engine . It supports Dynamically collect data from various data sources , And filter the data 、 analysis 、 Enrich 、 Unified format and other operations , then Store to the location specified by the user , Usually sent to ElasticSearch.

Logstash from Ruby Language writing , Running on the Java virtual machine （JVM) On , It's a powerful Data processing tools , Sure Realize data transmission 、 Format processing 、 Format output .Logstash It has powerful plug-in function , Commonly used in Log processing .

Filebeat：

Filebeat Lightweight open source log file data collector .

Filebeat It is usually installed on the client that needs to collect data Filebeat, and Specify directory and log format ,Filebeat Just Can quickly collect data , and Send to logstash To analyze , or Direct issue ElasticSearch Storage .

Filebeat Performance compared to JVM Upper logstash Obvious advantages , It's a replacement . Chang Ying be used for EFLK framework among .

Filebeat combination logstash Benefits ：

1. adopt Logstash With disk based adaptive buffering system , The system will absorb the incoming throughput , To lessen ElasticSearch Pressure to keep writing data
2. From other data sources （ Like databases ,S3 Object store or message delivery queue ） Extract from
3. Sending data to multiple destinations , for example S3,HDFS（Hadoop distributed file system ） Or write to a file
4. Use conditional data flow logic to form more complex processing pipelines

• cache / Message queue （redis、kafka、RabbitAQ etc. ）： Traffic peak shaving and buffering can be carried out for high concurrency log data , Such buffering can protect data from loss to a certain extent , You can also apply decoupling to the entire architecture .

Fluentd：

Fluentd Is a popular open source data collector . because logstash The disadvantage of being too heavy ,Logstash Low performance 、 More resource consumption and other problems , And then there's this Fluentd Appearance .

Fluentd Comparison logstash,Fluentd Easier to use 、 Less resource consumption 、 Higher performance , More efficient and reliable in data processing , Welcomed by enterprises , Become logstash An alternative to , Often applied to EFK Architecture .

Fluentd stay Kubernetes It is also commonly used in clusters EFK As a scheme for log data collection .

Fluentd stay Kubernetes In the cluster, it is generally through DaemonSet To run. , So that it's in every Kubernetes You can run one on a work node Pod.

Fluentd It gets the container log file 、 Filter and transform log data , And then pass the data to ElasticSearch colony , Index and store it in the cluster .

---

Ii. 、 Why use ELK

Logs mainly include system logs 、 Application logs and security logs . The system operation and maintenance personnel and developers can understand the software and hardware information of the server through the log 、 Check the errors in the configuration process and the causes of the errors . Regular log analysis can help you understand the load of the server , Performance security , So as to take measures to correct mistakes in time . Patrol the log regularly , Make countermeasures according to the error warning .

Often we use the log of a single machine grep、awk And other tools can basically achieve simple analysis , But when logs are distributed across different devices . If you manage hundreds of servers , You're still using the traditional method of logging in each machine in turn to look up the logs . Does this feel tedious and inefficient . We need to use centralized log management .

for example ： Open source syslog, Summarize the log collection on all servers . After centralized management of logs , Log statistics and retrieval has become a more cumbersome thing , Generally we use grep、awk and wc etc. Linux Command can realize retrieval and statistics , But for more demanding queries 、 Sorting and statistics requirements and large number of machines are still using this method, which is hard to avoid .

Generally, a large system is a distributed deployment architecture , Different service modules are deployed on different servers , When problems arise , Most situations need to be based on the key information exposed by the problem , Go to specific servers and service modules , Building a centralized log system , It can improve the efficiency of location problem .

---

3 、 The basic characteristics of complete log system

collect ： Be able to collect log data from multiple sources
transmission ： It can stably parse, filter and transmit log data to the storage system
Storage ： Store log data
analysis ： Support UI analysis
Warning ： Able to provide error reports , Monitoring mechanism

---

boss 、ELK How it works

（1） Deploy on all servers that need to collect logs Logstash, Or you can centralize the log management on the log server , Deploy on the log server Logstash.
（2）Logstash Collect the logs , Format the log and output it to ElasticSearch In a crowd .
（3）ElasticSearch Index and store the formatted data .
（4）Kibana from ES Query data in the cluster to generate charts , And display the front-end data .

summary ：logstash As a log collector , Collect data from a data source , And filter the data , format processing , And then leave it to ElasticSearch Storage ,kibana Visualize the log .

---

wu 、ELK Deployment of log analysis system

Environment configuration

host	operating system	IP Address	Tools / software package
node1	CentOS7	192.168.80.20	Elasticsearch/Kibana
node2	CentOS7	192.168.80.30	Elasticsearch
apache	CentOS7	192.168.80.50	httpd / Logstash

Experiment preparation
Turn off the firewall and system security mechanism

Log in and copy

systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0

hostnamectl set-hostname  Host name 

     

      
1.
      
2.
      
3.
      
4.
      
5.
     

1、 To configure elasticsearch Environmental Science

node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

echo '192.168.80.20 node1' >> /etc/hosts
echo '192.168.80.30 node2' >> /etc/hosts

java -version    # If not installed ,yum -y install java

     

      
1.
      
2.
      
3.
      
4.
     

2、 Deploy elasticsearch Software

node1（192.168.80.20）
node2（192.168.80.30）
（1） install elasticsearch—rpm package
Upload elasticsearch-5.5.0.rpm To /opt Under the table of contents

Log in and copy

cd /opt
rpm -ivh elasticsearch-5.5.0.rpm

# Load system services 
systemctl daemon-reload
systemctl enable elasticsearch.service

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

（2） change elasticsearch Master profile

Log in and copy

cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak

vim /etc/elasticsearch/elasticsearch.yml
#17 That's ok ; uncomment , modify ; The cluster name 
cluster.name: my-elk-cluster
#23 That's ok ; uncomment , modify ; Node name （node2 Modified into node2）
node.name: node1
#33 That's ok ; uncomment , modify ; Data storage path 
path.data: /data/elk_data
#37 That's ok ; uncomment , modify ; Log storage path 
path.logs: /var/log/elasticsearch
#43 That's ok ; uncomment , modify ; Don't lock memory at boot time 
bootstrap.memory_lock: false
#55 That's ok ; uncomment , modify ; Providing service binding IP Address ,0.0.0.0 For all addresses 
network.host: 0.0.0.0
#59 That's ok ; uncomment ; The listening port is 9200（ Default ）
http.port: 9200
#68 That's ok ; uncomment , modify ; Cluster discovery is realized by unicast , Specify the nodes to discover  node1、node2
discovery.zen.ping.unicast.hosts: ["node1", "node2"]

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
      
14.
      
15.
      
16.
      
17.
      
18.
      
19.
     

• Verification configuration

Log in and copy

grep -v "^#" /etc/elasticsearch/elasticsearch.yml

     

      
1.
     

（3） Create data storage path and authorize

Log in and copy

mkdir -p /data/elk_data
chown elasticsearch:elasticsearch /data/elk_data/

     

      
1.
      
2.
     

（4） start-up elasticsearch Is it successfully opened

Log in and copy

systemctl start elasticsearch
netstat -antp |grep 9200

     

      
1.
      
2.
     

（5） View node information

Log in and copy

http://192.168.80.20:9200
http://192.168.80.30:9200

     

      
1.
      
2.
     

（6） Verify cluster health status

Log in and copy

http://192.168.80.20:9200/_cluster/health?pretty
http://192.168.80.30:9200/_cluster/health?pretty

     

      
1.
      
2.
     

（7） View the cluster status

Log in and copy

http://192.168.80.20:9200/_cluster/state?pretty
http://192.168.80.30:9200/_cluster/state?pretty

     

      
1.
      
2.
     

3、 install elasticsearch-head plug-in unit

• install elasticsearch-head plug-in unit , be used for Management cluster

（1） Compilation and installation node Component dependency package
node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

yum -y install gcc gcc-c++ make

 Upload package  node-v8.2.1.tar.gz  To /opt
cd /opt
tar xzvf node-v8.2.1.tar.gz
cd node-v8.2.1/
./configure && make && make install

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
     

（2） install phantomjs（ The front frame ）
node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

 Upload package  phantomjs-2.1.1-linux-x86_64.tar.bz2  To /opt Under the table of contents 
cd /opt
tar jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/src/
cd /usr/local/src/phantomjs-2.1.1-linux-x86_64/bin
cp phantomjs /usr/local/bin

     

      
1.
      
2.
      
3.
      
4.
      
5.
     

（3） install elasticsearch-head（ Data visualization tool ）
node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

 Upload package  elasticsearch-head.tar.gz  To /opt
cd /opt
tar zxvf elasticsearch-head.tar.gz -C /usr/local/src/
cd /usr/local/src/elasticsearch-head/
npm install

     

      
1.
      
2.
      
3.
      
4.
      
5.
     

（4） Modify master profile
node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

vim /etc/elasticsearch/elasticsearch.yml
......
#--------- At the end of ; Add the following 
http.cors.enabled: true
http.cors.allow-origin: "*"

#--------- Parameter interpretation 
http.cors.enabled: true				# Enable cross domain access support , The default is  false
http.cors.allow-origin: "*"			# Specify the domain names and addresses allowed for cross domain access for all 


systemctl restart elasticsearch.service

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
     

（5） start-up elasticsearch-head

node1（192.168.80.20）
node2（192.168.80.30）

Log in and copy

 Must be after decompression  elasticsearch-head  Start the service in the directory , The process will read the  gruntfile.js  file , Otherwise, it may fail to start .
cd /usr/local/src/elasticsearch-head/
npm run start &

> [email protected] start /usr/local/src/elasticsearch-head
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100

elasticsearch-head  The listening port is  9100
netstat -natp |grep 9100

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
     

（6） Use elasticsearch-head Plug in to view cluster status

Log in and copy

http://192.168.80.20:9100
 stay Elasticsearch  Enter... In the following column 
http://192.168.80.20:9200

http://192.168.80.30:9100
 stay Elasticsearch  Enter... In the following column 
http://192.168.80.30:9200

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
     

（7） Create index
node1（192.168.80.20）
Create index as index-demo, The type is test

Log in and copy

curl -XPUT 'localhost:9200/index-demo/test/1?pretty&pretty' -H 'content-Type: application/json' -d '{"user":"zhangsan","mesg":"hello world"}'

     

      
1.
     

• Open the browser and enter the address , View index information

Log in and copy

http://192.168.80.20:9100

     

      
1.
     

• Click data to browse – Will find node1 The index created on is index-demo, The type is test, Relevant information

4、 install logstash

Collect logs and output to elasticsearch in

（1） install Apahce service （httpd）
apache（192.168.80.50）

Log in and copy

yum -y install httpd
systemctl start httpd

     

      
1.
      
2.
     

（2） install Java Environmental Science
apache（192.168.80.50）

Log in and copy

java -version        ### If not   install yum -y install java

     

      
1.
     

（3） install logstash
apache（192.168.80.50）

Log in and copy

 Upload logstash-5.5.1.rpm To /opt Under the table of contents 
cd /opt
rpm -ivh logstash-5.5.1.rpm

systemctl start logstash.service
systemctl enable logstash.service

# establish logstash Soft connection 
ln -s /usr/share/logstash/bin/logstash /usr/local/bin/

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
     

（4） test logstash command
apache（192.168.80.50）

Log in and copy

 Field description explains ：
-f   With this option you can specify logstash Configuration file for , Configure according to the configuration file logstash
-e   Followed by a string   The string can be treated as logstash Configuration of （ If it is ” ”, It is used by default stdin As input 、stdout As the output ）
-t   Test the configuration file for correctness , And then quit 

 Define input and output streams ：
 The input is standard input , The output is standard output （ Similar pipe ）
logstash -e 'input { stdin{} } output { stdout{} }'

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
     

• Use rubydebug Show detailed output ,codec For a codec

Log in and copy

logstash -e 'input { stdin{} } output { stdout{ codec=>rubydebug } }'

     

      
1.
     

• Use Logstash Write the information Elasticsearch in

Log in and copy

logstash -e 'input { stdin{} } output { elasticsearch { hosts=>["192.168.80.20:9200"] } }'

     

      
1.
     

• View index information

Log in and copy

 More logstash- date 
http://192.168.80.20:9100

     

      
1.
      
2.
     

• Click data browse to view the content of the response

（5） stay Apache Make docking configuration on the host
apache（192.168.80.50）

• Logstash The configuration file consists of three parts ：input、output、filter（ According to need ）

Log in and copy

chmod o+r /var/log/messages
ll /var/log/messages

vim /etc/logstash/conf.d/system.conf
input {
       file{
        path => "/var/log/messages"
        type => "system"
        start_position => "beginning"
        }
      }
output {
        elasticsearch {
          hosts => ["192.168.80.20:9200"]
          index => "system-%{+YYYY.MM.dd}"
          }
        }


systemctl restart logstash.service

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
      
14.
      
15.
      
16.
      
17.
      
18.
      
19.
      
20.
     

• View index information

Log in and copy

 More  system- date 
http://192.168.80.20:9100

     

      
1.
      
2.
     

5、 install kibana

node1（192.168.80.20）

Log in and copy

 Upload kibana-5.5.1-x86_64.rpm  To /opt Catalog 
cd /opt
rpm -ivh kibana-5.5.1-x86_64.rpm

cd /etc/kibana/
cp kibana.yml kibana.yml.bak

vim kibana.yml
#2 That's ok ; uncomment ;kibana Open port （ Default 5601）
server.port: 5601
#7 That's ok ; uncomment , modify ;kibana Listening address 
server.host: "0.0.0.0"
#21 That's ok ; uncomment , modify ; and elasticsearch Make connections 
elasticsearch.url: "http://192.168.80.20:9200"
#30 That's ok ; uncomment ; stay elasticsearch Add .kibana Indexes 
kibana.index: ".kibana"              				

systemctl start kibana.service 
systemctl enable kibana.service

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
      
14.
      
15.
      
16.
      
17.
      
18.
      
19.
     

• Access on Browser

Log in and copy

192.168.80.20:5601

# Create an index when logging in for the first time   name ：system-* （ This is the docking system log file ）, Then point to the bottom to come out create  Button to create 

     

      
1.
      
2.
      
3.
     

• Then click the top left corner Discover Button Will find system-* Information

• Then click the following host Lateral add You will find that the picture on the right is only Time and host Options. This is friendly

（6） docking Apache The host Apache Log files （ Access log 、 Error log ）
apache（192.168.80.50）

Log in and copy

cd /etc/logstash/conf.d/

vim apache_log.conf
input {
     file{
        path => "/etc/httpd/logs/access_log"
        type => "access"
        start_position => "beginning"
      }
     file{
        path => "/etc/httpd/logs/error_log"
        type => "error"
        start_position => "beginning"
      }
}
output {
    if [type] == "access" {
        elasticsearch {
          hosts => ["192.168.80.20:9200"]
          index => "apache_access-%{+YYYY.MM.dd}"
        }
    }
    if [type] == "error" {
        elasticsearch {
          hosts => ["192.168.80.20:9200"]
          index => "apache_error-%{+YYYY.MM.dd}"
        }
    }
}

/usr/share/logstash/bin/logstash -f apache_log.conf

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
      
14.
      
15.
      
16.
      
17.
      
18.
      
19.
      
20.
      
21.
      
22.
      
23.
      
24.
      
25.
      
26.
      
27.
      
28.
      
29.
      
30.
      
31.
     

Open input on the browser http://192.168.80.50, Manufacturing point access record

Open the browser Input http://192.168.80.20:9100/ View index information
Can find apache_error-2022.03.01 and apache_access-2022.03.01

Open the browser Input http://192.168.80.20:5601
Click on the bottom left corner to have a management Options —index patterns—create index pattern
Create separate apache_error-* and apache_access-* The index of

---

summary

There are still some imperfections in this framework , So we can continue to optimize 、 Extended architecture , For example, expand to efk framework .
efk Architecture is made up of elasticsearch+logstash+filebeat+kafka+kibana+redis constitute , among elasticsearch For indexing and storing data ;logstash For format conversion ;filebeat（ Lightweight file collection tools ） For log collection ;kafka（ Message queue , It can process hundreds of thousands of concurrent data per second ）+redis（ Caching services ） Used to resist high concurrency ;kibana For the display of front-end data .