当前位置:网站首页>Elk project monitoring platform deployment + deployment of detailed use (II)
Elk project monitoring platform deployment + deployment of detailed use (II)
2022-07-06 09:41:00 【wx5caecf2ed0645】
Because there may be some differences between different version numbers , So I'm here to ensure the smooth implementation of the next step . First confirm the version number and the deployment environment .
elasticsearch 5.5.1
logstash 5.5.1
kibana 5.5.1
These three servers are deployed in one server . Those with large business volume can consider elasticsearch Separate and cluster .
For convenience, the following are in es To represent the elasticsearc.
es + kibana Choose to use it for the convenience of being unfamiliar docker
stay linux Lower installation docker After execution :
docker pull docker.elastic.co/elasticsearch/elasticsearch:5.5.1
docker pull docker.elastic.co/kibana/kibana:5.5.1
logstash Use the source code to install :
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.1.tar.gz
cp logstash-5.6.1.tar.gz /usr/share/logstash.tar.gz
tar -zxvf logstash.tar.gz
cd logstash
./bin/logstash -e 'input { stdin {} } output { stdout {} }'
Test the installation for success
Now try starting es and kibana:
elasticsearch start-up :
docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" --name my-elastic -d docker.elastic.co/elasticsearch/elasticsearch:5.5.1
kibana start-up :
docker run -p 5601:5601 -e "ELASTICSEARCH_URL=http://localhost:9200" --name my-kibana --network host -d docker.elastic.co/kibana/kibana:5.5.1
At this time, if everything is implemented smoothly, you should be able to access kibana 了
The access address is deployed for you http://ip:5601 visit
Of course, there is no data at this time , Now on the project server Deploy data collection filebeat and metricbeat
1. Download these two files in the project server :
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.6.1-linux-x86_64.tar.gz
2. decompression
tar -zxvf *.tar.gz
3. To configure filebeat With ngxin For example, log
cd filebeat-5.6.1-linux-x86_64
vim nginx.yml
Copy the contents of the following file
filebeat.prospectors:
- input_type: log
paths:
- /var/log/nginx/access.log
document_type: nginx_access
# Then group and count the logs
fields:
level: debug
#Filebeat Start from the end of the file to monitor what's new in the file , Send each new line of file as an event in turn
tail_files: true
shipper:
tags: ['nginx-access']
# Project name and project service ip The address can be customized
tags: ["myserver", "101.110.56.78"]
output.logstash:
# logstash Server's ip Address
hosts: ["11.142.42.77:5044"]
shipper yes logstash Condition control is used when receiving logs
output.logstash To configure logstash Server's ip Address + Port number
start-up filebeat
./filebeat -e -c ./nginx.yml -d "publish"
To configure logstash
Get into logstash The installation directory
vim logstash.yml
input {
beats {
port => 5044
}
}
filter {
if [type] == "nginx_access" {
ruby {
init => "@kname = ['remote_addr','remote_user','time_local','request','status','body_bytes_sent','http_referer','http_user_agent','http_x_forwarded_for']"
code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('message').split(' | '))]))"
}
if [request] {
ruby {
init => "@kname = ['method','uri','verb']"
code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))]))"
}
if [uri] {
ruby {
init => "@kname = ['url_path','url_args']"
code => "event.append(LogStash::Event.new(Hash[@kname.zip(event.get('request').split('?'))]))"
}
kv {
prefix => "url_"
source => "url_args"
field_split => "& "
remove_field => [ "url_args","uri","request" ]
}
}
}
mutate {
convert => [ "body_bytes_sent" , "integer" ]
}
date {
match => [ "time_local", "dd/MMM/yyyy:hh:mm:ss Z" ]
locale => "en"
}
}
}
output {
if [type] == "nginx_access" {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
index => "nginx_access_%{+YYYY.MM.dd}"
user => "elastic"
password => "changeme"
}
if [status] != "200"{
exec {
command => "sh /root/sh/alarm.sh %{tags[0]} Found out BUG, Please deal with as soon as possible The server IP:%{tags[1]} request:%{request} Access status :%{status} Time :%{time_local}"
}
}
}
else {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
index => "metricbeat-%{+YYYY.MM.dd}"
user => "elastic"
password => "changeme"
}
}
}
It should be noted that :
1.nginx.conf You need to configure nginx The format of is convenient for logstash analytical
log_format main "$remote_addr | $remote_user | $time_local | $request | $status | $body_bytes_sent | $http_referer | $http_user_agent | $http_x_forwarded_for";
access_log /var/log/nginx/access.log main;
2. Need to install logstash plug-in unit exec To execute the script
3. Script alarm.sh The content is :
start-up logstash:
./logstash -f ./logstash.yml
边栏推荐
- 《ASP.NET Core 6框架揭秘》样章发布[200页/5章]
- Hero League rotation map automatic rotation
- [shell script] - archive file script
- 英雄联盟轮播图自动轮播
- Global and Chinese market of airport kiosks 2022-2028: Research Report on technology, participants, trends, market size and share
- Global and Chinese market of AVR series microcontrollers 2022-2028: Research Report on technology, participants, trends, market size and share
- Redis cluster
- What is an R-value reference and what is the difference between it and an l-value?
- [deep learning] semantic segmentation: paper reading: (CVPR 2022) mpvit (cnn+transformer): multipath visual transformer for dense prediction
- MapReduce instance (IX): reduce end join
猜你喜欢
Mapreduce实例(十):ChainMapReduce
CAP理论
Use of activiti7 workflow
Le modèle sentinelle de redis
DCDC power ripple test
Redis geospatial
[Yu Yue education] reference materials of complex variable function and integral transformation of Shenyang University of Technology
Persistence practice of redis (Linux version)
Blue Bridge Cup_ Single chip microcomputer_ Measure the frequency of 555
Heap (priority queue) topic
随机推荐
【深度学习】语义分割:论文阅读:(CVPR 2022) MPViT(CNN+Transformer):用于密集预测的多路径视觉Transformer
Several ways of MySQL database optimization (pen interview must ask)
五月刷题26——并查集
Use of activiti7 workflow
Redis分布式锁实现Redisson 15问
Mapreduce实例(六):倒排索引
Le modèle sentinelle de redis
基于B/S的影视创作论坛的设计与实现(附:源码 论文 sql文件 项目部署教程)
Redis cluster
IDS cache preheating, avalanche, penetration
Segmentation sémantique de l'apprentissage profond - résumé du code source
Leetcode:608 tree node
MapReduce instance (VIII): Map end join
Une grande vague d'attaques à la source ouverte
Heap (priority queue) topic
五月刷题27——图
MySQL数据库优化的几种方式(笔面试必问)
Day 5 of MySQL learning
Kratos ares microservice framework (II)
Oom happened. Do you know the reason and how to solve it?