当前位置：网站首页>Beats (filebeat, metricbeat), kibana, logstack tutorial of elastic stack
Beats (filebeat, metricbeat), kibana, logstack tutorial of elastic stack
2022-07-02 09:25:00 【niceyz】
If you haven't heard of Elastic Stack, Then you must have heard of ELK, actually ELK It is the abbreviation of three kinds of software , Namely Elasticsearch、Logstash、Kibana form , In the process of development , There are new members Beats The addition of , So it forms Elastic Stack. So ,ELK It's the old name ,Elastic Stack It's a new name .
The whole department Elastic Stack The technology stack includes ：
As can be seen from the figure above Beats It doesn't mean a single technology , It refers to a series of technologies in general , The collection ability is more lightweight and powerful , And has gradually replaced Logstash The status of .
https://www.elastic.co/cn/beats/
1. Beats brief introduction
2. Beats And Filebeat
2.1 framework
Used to monitor 、 Collect server log files .Harvester： harvester Spooler： Scroll , Transmit data downstream
2.2 Deploy and run
download : https://www.elastic.co/downloads/beats This article downloads ：filebeat-7.10.0-linux-x86_64.tar.gz
Extract to the specified directory ：tar -zxvf filebeat-7.10.0-linux-x86_64.tar.gz -C ../opt/
Enter into filebeat Catalog ：cd /opt/filebeat-7.10.0-linux-x86_64/
2.2.1 Read standard input
# Create the following configuration file vim yztest.yml Notice that there are spaces
filebeat.inputs:
- type: stdin The standard input
enabled: true Enable input
setup.template.settings:
index.number_of_shards: 3 Appoint ES Number of index partitions , Now I don't need
output.console: Output to console
pretty: true Output beautification
enable: true Output enable
./filebeat -e -c yztest.yml
2.3 Read the file
filebeat.inputs:
- type: log The input type is log file
enabled: true Enable input
paths:
- /opt/test/logs/*.log Log path to collect
setup.template.settings:
index.number_of_shards: 3 Appoint ES Number of index partitions , Now I don't need
output.console: Output to console
pretty: true Output beautification
enable: true Output enable
./filebeat -e -c yztest-log.yml
It can be seen that , An update to the log file has been detected , The updated content will be read immediately , And output to the console .
2.4 Custom field , Define the logs to be collected tab Tags are used to distinguish where they come from
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/test/logs/*.log
tags: ["fl-carloan-web"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: fl-carloan-web
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3
output.console:
pretty: true
enable: true
2.5 Output to Elasticsearch vim yztest-log.yml
- type: log # The input type is log
enabled: true
paths: # Log path to collect
- /opt/test/logs/*.log
tags: ["fl-carloan-web"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: fl-carloan-web
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3 # Appoint elasticsearch Number of partitions in the index
setup.template.pattern: "filebeat-*"
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]
2.6 FileBeat working principle
Filebeat It consists of two main components ：prospector（ Prospector ） and harvester（ harvester ）.
harvester：
Responsible for reading the contents of a single file .
If the file is deleted or renamed while reading ,Filebeat Will continue to read the file .
prospector：
prospector Responsible for managing the harvester And find the source of all the files to read .
If the input type is log , The finder will find all the files that match the path , And start one for each file harvester.
Filebeat There are currently two kinds of prospector type ：log and stdin.
Filebeat How to keep the status of the file （ Namely Filebeat Where to start reading after downtime ）：
Filebeat Save the status of each file and often refresh the status to the registration file on disk （ The file status is recorded in filebeat-7.10.0/data/registry/filebeat/log.json In file ）.
This state is used to remember harvester The last offset being read , And make sure all log lines are sent .
If the output （ for example Elasticsearch or Logstash） cannot access ,Filebeat Will track the last line sent , And continue to read the file when the output is available again .
stay Filebeat Runtime , Every prospector The file status information will also be saved in memory , When rebooting Filebeat when , The data of the registration file will be used to rebuild the file state ,Filebeat Each one harvester Continue reading from the last offset saved .
cat log.json Record the read log file and the last offset
Start command ：
2.7 Read Nginx Log files ,filebeat Integrate nginx Do a test
Brief introduction Nginx Log analysis system
1.1、 Project requirements
Nginx It's a very good web The server , Often nginx The service will serve as the access portal to the project , that ,nginx It is very important to ensure the performance of the system , If nginx Problems in the operation of the project will have a greater impact on the project , therefore , We need to be right about nginx There are monitoring measures for the operation of , Master in real time nginx Operating condition , Then it needs to be collected nginx Operation index and analysis of nginx The running log of .
1.2、 The business process
explain ：
1. adopt Beats collection Nginx Index data and log data of
2. Beats Collect the data and send it to Elasticsearch in
3. Kibana Read data for analysis
4. User pass Kibana To view the analysis report
filebeat Read nginx Log files
# Writing configuration files cd /opt/filebeat-7.10.0/ vim yztest-nginx.yml
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/*.log
tags: ["nginx"]
setup.template.settings:
index.number_of_shards: 3 # Appoint ES Number of partitions in the index
output.elasticsearch: # Appoint ES Configuration of
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]
You can see , stay message We've got nginx Log , however , The content is not processed , Just read the original data , So it's not good for our later operation , You can use the built-in module Let's deal with it .
2.8 Module
In front, you want to read and process the log data manually , Such as manual parsing message The contents of a string . Actually , stay Filebeat in , A large number of Module（ Use module analysis message character string , Like tools ）, Can simplify our configuration , You can use it directly , as follows ： command ./filebeat modules list
./filebeat modules disable nginx # Ban
cd modules.d/ ll see
vim nginx.yml
- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/usr/local/nginx/logs/access.log*"] # The log ends by day
# Error logs
error:
enabled: true
var.paths: ["/usr/local/nginx/logs/error.log*"] # The log ends by day
filebeat.inputs:
tags: ["nginx"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: nginx
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3 # Appoint elasticsearch Number of partitions in the index
output.elasticsearch: # Specify output to elasticsearch,hosts by es The cluster address
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
start-up ./filebeat -e -c yztest-nginx-module.yml
If there is an error , Solve it in the following way , There is no error when starting here .
How many requests nginx, Give Way filebeat Read nginx Log file information and refresh to elasticsearch in , stay elasticsearch You can see , Use nginx-modul after , The returned information is a little thinner than the previous disassembly .
Mainly collect data of indicators
Collect index data of operating system or application service regularly
Store in Elasticsearch in , Real time analysis
Metricbeat Yes 2 Part of it is made up of , Part of it is Module, The other part is Metricset.
Module： Collected objects , Such as ：mysql、redis、nginx、 Operating system, etc ;
Metricset： Collect a collection of indicators , Such as ：cpu、memory（ Memory ）、network（ The Internet ） etc. ;
1. tar -zxvf metricbeat-7.10.0-linux-x86_64.tar.gz -C /opt/ mv metricbeat-7.10.0-linux-x86_64/ metricbeat-7.10.0
2. cd metricbeat-7.10.0
3. vim metricbeat.yml Modify only elasticsearch Output path , The rest remains the same ,es The number of slices can be changed, but not changed ：
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]
By default, the system indicator collection is on ：
stay nginx in , Need to turn on status query , To query the index data
# recompile nginx
cd /opt/nginx-1.19.4/ perform ： ./configure --prefix=/opt/context/nginx --with-http_ssl_module --with-http_stub_status_module
Execute first ：make In execution ：make install
cd /usr/local/nginx/sbin perform ： ./nginx -V # Query version information nginx Status module installed 
# To configure nginx
cd /usr/local/nginx/conf vim nginx.conf add to ：
location /nginx-status {
stub_status on;
access_log off;
}
restart nginx ： cd /usr/local/nginx/sbin/ perform ： ./nginx -s reload ( If the order is not easy to use , Just stop nginx: ./nginx -s stop Start up ./nginx)
Access the just configured /nginx-status
The result shows that ：
Active connections： Number of active connections being processed
server accepts handled requests
first server Express Nginx It's been started up until now 15 A connection
the second accepts Express Nginx It has been successfully created up to now 15 The second handshake
Third handled requests In total, we have dealt with 23 Requests
Number of requests lost = The number of handshakes - The number of connections , As you can see, no requests have been lost so far
Reading: 0 Writing: 1 Waiting: 1
Reading：Nginx Read... To the client Header Number of messages
Writing：Nginx Return to the client Header Number of messages
Waiting：Nginx Finished processing resident link waiting for next request instruction （ Turn on keep-alive Under the circumstances , This value is equal to Active - (Reading+Writing)
- module: nginx
#metricsets:
# - stubstatus
period: 10s
# Nginx hosts
#hosts: ["http://127.0.0.1"]
hosts: ["http://192.168.226.30"]
# Path to server status. Default server-status
#server_status_path: "server-status"
server_status_path: "nginx-status"
#username: "user"
#password: "secret"
Kibana Is an open source data analysis and visualization platform , It is Elastic Stack One of the members , Designed for and Elasticsearch Collaboration . You can use Kibana Yes Elasticsearch Search the data in the index 、 see 、 interactions . You can easily use charts 、 Tables and maps provide diversified analysis and presentation of data .
4.1 configuration setup
xpack.security.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.reporting.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.reporting.capture.browser.chromium.disableSandbox: false # Turn off the sandbox
After creating the index , After a while, you can see the data of performance indicators on the dashboard ：
I added a few casually , That's too much , Here is the will es An operation of data visualization in
Re metricbeat Run up ： ./metricbeat -e Let it constantly generate data .
stay kibana Click on the dashboard
These are the newly installed instrument panels
find [Metricbeat System] Host Services Overview , Click in to see some data information of the system
边栏推荐
- 【Go实战基础】gin 如何绑定与使用 url 参数
- 深入剖析JVM是如何执行Hello World的
- AMQ6126问题解决思路
- Win10 uses docker to pull the redis image and reports an error read only file system: unknown
- Knowledge points are very detailed (code is annotated) number structure (C language) -- Chapter 3, stack and queue
- Break the cocoon | one article explains what is the real cloud primordial
- [go practical basis] how to install and use gin
- 【Go实战基础】gin 高效神器，如何将参数绑定到结构体
- idea查看字节码配置
- Microservice practice | declarative service invocation openfeign practice
猜你喜欢
Avoid breaking changes caused by modifying constructor input parameters
![[go practical basis] how can gin get the request parameters of get and post](/img/fd/66074d157d93bcf20a5d3b37da9b3e.png)
[go practical basis] how can gin get the request parameters of get and post
【Go实战基础】gin 如何自定义和使用一个中间件
Say goodbye to 996. What are the necessary plug-ins in idea?
Chrome浏览器标签管理插件–OneTab
![[go practical basis] how to verify request parameters in gin](/img/de/50db131d6993e5d955e3416c667c4c.png)
[go practical basis] how to verify request parameters in gin
一篇详解带你再次重现《统计学习方法》——第二章、感知机模型
Programmers with ten years of development experience tell you, what core competitiveness do you lack?
cmd窗口中中文呈现乱码解决方法
Chrome user script manager tempermonkey monkey
随机推荐
Knife4j 2. Solution to the problem of file control without selection when uploading x version files
Matplotlib剑客行——没有工具用代码也能画图的造型师
"Interview high frequency question" is 1.5/5 difficult, and the classic "prefix and + dichotomy" application question
Chrome browser plug-in fatkun installation and introduction
Redis installation and deployment (windows/linux)
Oracle修改表空间名称以及数据文件
C language implementation of mine sweeping game
[staff] time sign and note duration (full note | half note | quarter note | eighth note | sixteenth note | thirty second note)
Oracle delete tablespace and user
西瓜书--第五章.神经网络
Long summary (code with comments) number structure (C language) -- Chapter 4, string (Part 1)
I've taken it. MySQL table 500W rows, but someone doesn't partition it?
Redis zadd导致的一次线上问题排查和处理
京东面试官问：LEFT JOIN关联表中用ON还是WHERE跟条件有什么区别
In depth analysis of how the JVM executes Hello World
深入剖析JVM是如何执行Hello World的
盘点典型错误之TypeError: X() got multiple values for argument ‘Y‘
VIM operation command Encyclopedia
Cloudreve自建云盘实践，我说了没人能限制得了我的容量和速度
十年开发经验的程序员告诉你，你还缺少哪些核心竞争力？