Elk log collection system deployment

Log is a very extensive concept in computer system , Any program can output logs ： Operating system kernel 、 Various application servers and so on . Contents of the log 、 It's also different in scale and use , It's hard to generalize .

Web The log contains a lot of people —— Mainly the information that the product analyst will be interested in , The simplest , We can get the information about each type of page in the website PV value （PageView, Page visits ）、 Independent IP Count （ After weight removal IP Number ） etc. ; A little more complicated , You can calculate the keyword ranking list retrieved by the user 、 The page with the highest user stay time, etc ; More complicated , Build ad Click model 、 Analyze user behavior characteristics, etc .

Today, I'd like to introduce a log analysis tool :ELK

ELK from Elasticsearch、Logstash and Kibana Three components ;

Elasticsearch Is based on JSON Distributed search and analysis engine , Designed for horizontal scaling 、 Designed for high availability and easy management

Logstash Is an open source server-side data processing pipeline , Can simultaneously Collect data from multiple sources 、 Conversion data , Then send the data to your favorite “ The repository ” in .（ Our repository, of course, is Elasticsearch.）

Kibana Be able to present data in the form of charts , And has an extensible user interface , For all-round configuration and management Elastic Stack.

Today's experiment is : adopt ELK Analyze all online Nginx Access log .

One 、 Test topology

Two 、 Software packages get

Nginx download http://nginx.org/en/download.html

Redis download https://redis.io/

Elasticsearch logstash kibana download https://www.elastic.co/downloads

3、 ... and , Start deployment

3.1） Business machine deployment A

Business machine :192.168.1.242/24

OS：rhel6.5

Software involved :nginx+logstash+redis+jdk

Package preparation ： Download the software package according to the above prompts

[[email protected] opt]# ls

jdk-8u144-linux-x64.rpm logstash-5.5.1.tar.gz nginx-1.13.4.tar.gz redis-4.0.1.tar.gz

3.1.1） install JDK

[[email protected] opt]# rpm -ivh jdk-8u144-linux-x64.rpm

Preparing... ####################################### [100%]

1:jdk1.8.0_144 ######################################## [100%]

Unpacking JAR files...

tools.jar...

plugin.jar...

javaws.jar...

deploy.jar...

rt.jar...

jsse.jar...

charsets.jar...

localedata.jar...

Set up java environment variable

[[email protected] opt]# vim /root/.bash_profile Add the following content at the end

JAVA_HOME=/usr/java/jdk1.8.0_144

PATH=JAVA_HOME/bin:PATH:

CLASSPATH=.:JAVA_HOME/lib/tools.jar:JAVA_HOME/lib/dt.jar

export PATH JAVA_HOME CLASSPATH CATALINA_HOME

Validate the configuration and verify

[[email protected] opt]# source /root/.bash_profile

[[email protected] opt]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

3.1.2） install redis

[[email protected] opt]# tar xf redis-4.0.1.tar.gz

[[email protected] opt]# cd redis-4.0.1

[[email protected] redis-4.0.1]# make

[[email protected] redis-4.0.1]# make install

To configure redis

[[email protected] redis-4.0.1]# sed -i -r '/^(bind)/s/127.0.0.1/0.0.0.0/' redis.conf

[[email protected] redis-4.0.1]# sed -i -r '/^(daemonize)/s/no/yes/' redis.conf

start-up redis

[[email protected] redis-4.0.1]# redis-server redis.conf

5789:C 30 Aug 11:09:58.584 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

5789:C 30 Aug 11:09:58.584 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=5789, just started

5789:C 30 Aug 11:09:58.584 # Configuration loaded

Verify startup

[[email protected] redis-4.0.1]# lsof -i :6379

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

redis-ser 5790 root 6u IPv4 18672 0t0 TCP *:6379 (LISTEN)

3.1.3） install nginx

[[email protected] opt]# tar xf nginx-1.13.4.tar.gz

[[email protected] opt]# cd nginx-1.13.4

[[email protected] nginx-1.13.4]# yum -y install pcre-devel zlib-devel

[[email protected] nginx-1.13.4]# ./configure --prefix=/usr/local/nginx

[[email protected] nginx-1.13.4]# make

[[email protected] nginx-1.13.4]# make install

modify nginx The configuration file , Redefinition log_format With json Format output log to access.log

[[email protected] nginx-1.13.4]# cd /usr/local/nginx/conf/

[[email protected] conf]# vim nginx.conf

start-up nginx And verify

[[email protected] conf]# /usr/local/nginx/sbin/nginx

[[email protected] conf]# lsof -i :80

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

nginx 10765 root 6u IPv4 24510 0t0 TCP *:http (LISTEN)

nginx 10766 nobody 6u IPv4 24510 0t0 TCP *:http (LISTEN)

A visit nginx Verify that the log format is correct

3.1.4） install logstash

[[email protected] opt]# tar xf logstash-5.5.1.tar.gz -C /usr/local/

[[email protected] opt]# cd /usr/local/logstash-5.5.1/

[[email protected] logstash-5.5.1]# mkdir conf.d

[[email protected] logstash-5.5.1]# vim conf.d/nginx_to_redis

input {

file {

path => ["/usr/local/nginx/logs/access.log"]

type => "nginx_log"

codec => json

}

}

output {

redis{

host => "192.168.1.242"

key => 'logstash:redis'

data_type => 'channel'

port => '6379'

}

stdout {

codec => rubydebug

}

}

start-up logstash And test whether the collection is successful nginx Log to redis

[[email protected] ~]# /usr/local/logstash-5.5.1/bin/logstash -f /usr/local/logstash-5.5.1/conf.d/nginx_to_redis

View startup log

[[email protected] logstash-5.5.1]# tailf /usr/local/logstash-5.5.1/logs/logstash-plain.log

Test log collection

Logstash Collect log output

Turn on redis monitor

3.2） Business machine deployment B

Business machine :192.168.1.241/24

OS：rhel6.5

Software involved :elasticsearch+logstash+kibana

[[email protected] opt]# ls

elasticsearch-5.5.1.rpm

kibana-5.5.1-x86_64.rpm

jdk-8u144-linux-x64.rpm

logstash-5.5.1.tar.gz

3.2.1） install jdk

Reference resources 242 Set up

3.2.2） install elasticsearch

[[email protected] opt]# rpm -ivh elasticsearch-5.5.1.rpm

warning: elasticsearch-5.5.1.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY

Preparing... ######################################## [100%]

Creating elasticsearch group... OK

Creating elasticsearch user... OK

1:elasticsearch ######################################## [100%]

### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig

sudo chkconfig --add elasticsearch

### You can start elasticsearch service by executing

sudo service elasticsearch start

To configure elasticsearch

[[email protected] opt]# sed -i -r '/^(#network.host:)/cnetwork.host: 0.0.0.0' /etc/elasticsearch/elasticsearch.yml

[[email protected] opt]# sed -i -r '/^(#http.port:)/chttp.port: 9200' /etc/elasticsearch/elasticsearch.yml

[[email protected] opt]# sed -i -r '/^(#bootstrap.memory_lock:)/cbootstrap.memory_lock: falsenbootstrap.system_call_filter: false' /etc/elasticsearch/elasticsearch.yml

Optimize the system

[[email protected] opt]# vim /etc/security/limits.conf Append at the end

elasticsearch soft nproc 10240

elasticsearch hard nproc 10240

* soft nofile 65540

* hard nofile 65540

Restart the computer to take effect

start-up elasticsearch

[[email protected] opt]# /etc/init.d/elasticsearch start

Starting elasticsearch: [ OK ]

verification

3.2.3） install logstash

[[email protected] opt]# tar xf logstash-5.5.1.tar.gz -C /usr/local/

[[email protected] opt]# cd /usr/local/logstash-5.5.1/

[[email protected] logstash-5.5.1]# mkdir conf.d

[[email protected] logstash-5.5.1]# vim conf.d/redis_to_elk

input {

redis {

port => "6379"

host => "192.168.1.242"

data_type => "channel"

key => "logstash:redis"

type => "redis-input"

}

}

output {

elasticsearch {

hosts => "192.168.1.241"

index => "logstash-%{+YYYY.MM.dd}"

action => "index"

}

stdout {

codec => rubydebug

}

}

start-up logstash

[[email protected] logstash-5.5.1]#./bin/logstash -f conf.d/redis_to_elk

Access the test data once to see if there is redis Write to elk

3.2.4） install kibana

[[email protected] opt]# rpm -ivh kibana-5.5.1-x86_64.rpm

warning: kibana-5.5.1-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY

Preparing... ####################################### [100%]

1:kibana ####################################### [100%]

Modify... In the configuration file

[[email protected] opt]# sed -i -r '/^(#server.host:)/cserver.host: "0.0.0.0"' /etc/kibana/kibana.yml

[[email protected] opt]# /etc/init.d/kibana start

kibana started

Verify startup

[[email protected] opt]# netstat -ntpl |grep 5601

tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1993/node

Test through browser