当前位置：网站首页>Beats (filebeat, metricbeat), kibana, logstack tutorial of elastic stack

Beats (filebeat, metricbeat), kibana, logstack tutorial of elastic stack

2022-07-02 09:25:00 【niceyz】

If you haven't heard of Elastic Stack, Then you must have heard of ELK, actually ELK It is the abbreviation of three kinds of software , Namely Elasticsearch、Logstash、Kibana form , In the process of development , There are new members Beats The addition of , So it forms Elastic Stack. So ,ELK It's the old name ,Elastic Stack It's a new name .

The whole department Elastic Stack The technology stack includes ：

As can be seen from the figure above Beats It doesn't mean a single technology , It refers to a series of technologies in general , The collection ability is more lightweight and powerful , And has gradually replaced Logstash The status of .

https://www.elastic.co/cn/beats/

1. Beats brief introduction

2. Beats And Filebeat

2.1 framework

Used to monitor 、 Collect server log files .Harvester： harvester Spooler： Scroll , Transmit data downstream

2.2 Deploy and run

download : https://www.elastic.co/downloads/beats This article downloads ：filebeat-7.10.0-linux-x86_64.tar.gz
Extract to the specified directory ：tar -zxvf filebeat-7.10.0-linux-x86_64.tar.gz -C ../opt/
Enter into filebeat Catalog ：cd /opt/filebeat-7.10.0-linux-x86_64/

2.2.1 Read standard input

# Create the following configuration file vim yztest.yml Notice that there are spaces

filebeat.inputs:
- type: stdin The standard input
  enabled: true Enable input
setup.template.settings:
  index.number_of_shards: 3   Appoint ES Number of index partitions , Now I don't need
output.console: Output to console
  pretty: true Output beautification
  enable: true Output enable

# start-up filebeat

./filebeat -e -c yztest.yml

# Input hello The operation results are as follows ：

2.3 Read the file

# Configuration read file entry yztest-log.yml Notice that there are spaces

filebeat.inputs:
- type: log The input type is log file
enabled: true Enable input
  paths:
       - /opt/test/logs/*.log Log path to collect
setup.template.settings:
   index.number_of_shards: 3 Appoint ES Number of index partitions , Now I don't need
output.console:                                   Output to console
   pretty: true Output beautification
   enable: true Output enable

# start-up filebeat

./filebeat -e -c yztest-log.yml

# Open another window , stay /opt/test/logs Create a.log file , And write hello

# Observe filebeat Output

It can be seen that , An update to the log file has been detected , The updated content will be read immediately , And output to the console .

2.4 Custom field , Define the logs to be collected tab Tags are used to distinguish where they come from

# Configuration read file entry yztest-log.yml

filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/test/logs/*.log
tags: ["fl-carloan-web"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: fl-carloan-web
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3
output.console:
pretty: true
enable: true

# start-up filebeat

./filebeat -e -c yztest-log.yml

stay a.log Add new content ：echo '1234' >> /opt/test/logs/a.log

# Execution effect

2.5 Output to Elasticsearch vim yztest-log.yml

filebeat.inputs:
- type: log # The input type is log
enabled: true
paths: # Log path to collect
- /opt/test/logs/*.log
tags: ["fl-carloan-web"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: fl-carloan-web
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3 # Appoint elasticsearch Number of partitions in the index

setup.template.name: "filebeat" # If you specify an index , You need to set a template
setup.template.pattern: "filebeat-*"

output.elasticsearch: # Specify output to elasticsearch
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]

index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}" # catalog index filebeat-7.10.0-2020-12-02

https://www.elastic.co/guide/en/beats/filebeat/7.10/elasticsearch-output.html

# start-up filebeat

./filebeat -e -c yztest-log.yml

stay a.log Add new content ：echo 'filebeat To collect ' >> /opt/test/logs/a.log

Go to ES see , You can see ES Has been received filebeat Push the data ：

2.6 FileBeat working principle

Filebeat It consists of two main components ：prospector（ Prospector ） and harvester（ harvester ）.
harvester：
Responsible for reading the contents of a single file .
If the file is deleted or renamed while reading ,Filebeat Will continue to read the file .
prospector：
prospector Responsible for managing the harvester And find the source of all the files to read .
If the input type is log , The finder will find all the files that match the path , And start one for each file harvester.
Filebeat There are currently two kinds of prospector type ：log and stdin.
Filebeat How to keep the status of the file （ Namely Filebeat Where to start reading after downtime ）：
Filebeat Save the status of each file and often refresh the status to the registration file on disk （ The file status is recorded in filebeat-7.10.0/data/registry/filebeat/log.json In file ）.
This state is used to remember harvester The last offset being read , And make sure all log lines are sent .
If the output （ for example Elasticsearch or Logstash） cannot access ,Filebeat Will track the last line sent , And continue to read the file when the output is available again .
stay Filebeat Runtime , Every prospector The file status information will also be saved in memory , When rebooting Filebeat when , The data of the registration file will be used to rebuild the file state ,Filebeat Each one harvester Continue reading from the last offset saved .

cat log.json Record the read log file and the last offset

Start command ：

./filebeat -e -c yztest.yml

./filebeat -e -c yztest.yml -d "publish"

# Parameter description

-e: Output to standard output , Default output to syslog and logs Next

-c: Specify profile

-d "publish" : Output debug Information （ In the output to ES In the middle of the day , Use debug Parameters , Can be in filebeat Console view debug Information ）

2.7 Read Nginx Log files ,filebeat Integrate nginx Do a test

Brief introduction Nginx Log analysis system

1.1、 Project requirements
Nginx It's a very good web The server , Often nginx The service will serve as the access portal to the project , that ,nginx It is very important to ensure the performance of the system , If nginx Problems in the operation of the project will have a greater impact on the project , therefore , We need to be right about nginx There are monitoring measures for the operation of , Master in real time nginx Operating condition , Then it needs to be collected nginx Operation index and analysis of nginx The running log of .
1.2、 The business process

explain ：
1. adopt Beats collection Nginx Index data and log data of
2. Beats Collect the data and send it to Elasticsearch in
3. Kibana Read data for analysis
4. User pass Kibana To view the analysis report

Deployment installation Nginx

1. Download and upload to the server http://nginx.org/en/download.html

2. decompression tar -zxvf nginx-1.19.4.tar.gz -C /opt/

3. Download dependency yum install -y gcc gcc-c++ zlib-devel pcre pcre-devel openssl openssl-devel

4. compile cd nginx-1.19.4/ perform ：./configure Wait for the configuration to complete , Execute after configuration ：make install Waiting for compilation and installation

5. start-up cd /usr/local/nginx/sbin/ ./nginx start-up nginx

6. Browser access page , Default 80 port .http://192.168.226.30/

6. Check the log tail -f /usr/local/nginx/logs/access.log Refresh the page and there will be a request log

filebeat Read nginx Log files

# Writing configuration files cd /opt/filebeat-7.10.0/ vim yztest-nginx.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
     - /usr/local/nginx/logs/*.log
  tags: ["nginx"]
setup.template.settings:
  index.number_of_shards: 3 # Appoint ES Number of partitions in the index
output.elasticsearch: # Appoint ES Configuration of
  hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]

# start-up ( First put the previous es in filebeat Data deletion , Convenient for this check )

./filebeat -e -c yztest-nginx.yml

After starting , Can be in Elasticsearch See index and view data in ：

You can see , stay message We've got nginx Log , however , The content is not processed , Just read the original data , So it's not good for our later operation , You can use the built-in module Let's deal with it .

2.8 Module

In front, you want to read and process the log data manually , Such as manual parsing message The contents of a string . Actually , stay Filebeat in , A large number of Module（ Use module analysis message character string , Like tools ）, Can simplify our configuration , You can use it directly , as follows ： command ./filebeat modules list

You can see , Built in a lot of module, But none of them are enabled , If it needs to be enabled, it needs to be enable operation , Enabled nginx module For example ：

./filebeat modules enable nginx # start-up
./filebeat modules disable nginx # Ban

2.7.1、nginx module To configure

cd modules.d/ ll see

vim nginx.yml

- module: nginx
# Access logs
access:
enabled: true
var.paths: ["/usr/local/nginx/logs/access.log*"] # The log ends by day

# Error logs
error:
enabled: true
var.paths: ["/usr/local/nginx/logs/error.log*"] # The log ends by day

2.7.2、 To configure fifilebeat

# cp yztest-nginx.yml yztest-nginx-module.yml vim yztest-nginx-module.yml Comment out the input , Configure to modules Input

filebeat.inputs:
tags: ["nginx"] # Add custom tag, For subsequent processing
fields: # Add custom fields , Explain where the log comes from
from: nginx
fields_under_root: true #true To add to the root node ,false To add to child nodes
setup.template.settings:
index.number_of_shards: 3 # Appoint elasticsearch Number of partitions in the index
output.elasticsearch: # Specify output to elasticsearch,hosts by es The cluster address
hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

start-up ./filebeat -e -c yztest-nginx-module.yml

If there is an error , Solve it in the following way , There is no error when starting here .

ERROR fileset/factory.go:142 Error loading pipeline: Error loading pipeline for fileset nginx/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:

sudo bin/elasticsearch-plugin install ingest-user-agent

sudo bin/elasticsearch-plugin install ingest-geoip

# solve ： Need to be in Elasticsearch Install in ingest-user-agent、ingest-geoip plug-in unit

# among ,ingest-user-agent.tar、ingest-geoip.tar Unzip to plugins Next

#ingest-geoip-conf.tar Unzip to config Next houd

How many requests nginx, Give Way filebeat Read nginx Log file information and refresh to elasticsearch in , stay elasticsearch You can see , Use nginx-modul after , The returned information is a little thinner than the previous disassembly .

Other Module Refer to the official documents for the usage of ：

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html

3、Metricbeat https://www.elastic.co/cn/beats/metricbeat

Mainly collect data of indicators
Collect index data of operating system or application service regularly
Store in Elasticsearch in , Real time analysis

3.1 Metricbeat form

Metricbeat Yes 2 Part of it is made up of , Part of it is Module, The other part is Metricset.
Module： Collected objects , Such as ：mysql、redis、nginx、 Operating system, etc ;
Metricset： Collect a collection of indicators , Such as ：cpu、memory（ Memory ）、network（ The Internet ） etc. ;

With Redis Module For example ：

3.2 Metricbeat Deploy and collect system indicators

1. tar -zxvf metricbeat-7.10.0-linux-x86_64.tar.gz -C /opt/ mv metricbeat-7.10.0-linux-x86_64/ metricbeat-7.10.0
2. cd metricbeat-7.10.0
3. vim metricbeat.yml Modify only elasticsearch Output path , The rest remains the same ,es The number of slices can be changed, but not changed ：

hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200"]

metricbeat.config.modules:

path: ${path.config}/modules.d/*.yml

reload.enabled: false

setup.template.settings:

index.number_of_shards: 3

index.codec: best_compression

setup.kibana:

output.elasticsearch:

hosts: ["192.168.226.30:9200","192.168.226.31:9200","192.168.226.32:9200" ]

processors:

- add_host_metadata: ~

- add_cloud_metadata: ~

By default, the system indicator collection is on ：

# start-up

./metricbeat -e After starting, the indicator data of the system will be automatically collected and sent to esasticsearch in , Because in metricbeat.yml Is used in modlues, And in moddules.d Directory , System modul It's on .

stay esasticsearch Can be seen in , from metricbeat The system indicator data collected and pushed has come ：

3.3 Metricbeat Of Module, Basic and Filebeat Of Module Use the same

./metricbeat modules list # view list

3.4 Nginx Module

stay nginx in , Need to turn on status query , To query the index data

# recompile nginx
cd /opt/nginx-1.19.4/ perform ： ./configure --prefix=/opt/context/nginx --with-http_ssl_module --with-http_stub_status_module
Execute first ：make In execution ：make install
cd /usr/local/nginx/sbin perform ： ./nginx -V # Query version information nginx Status module installed

# To configure nginx
cd /usr/local/nginx/conf vim nginx.conf add to ：

location /nginx-status {
stub_status on;
access_log off;
}

restart nginx ： cd /usr/local/nginx/sbin/ perform ： ./nginx -s reload ( If the order is not easy to use , Just stop nginx: ./nginx -s stop Start up ./nginx)

Access the just configured /nginx-status

The result shows that ：

Active connections： Number of active connections being processed
server accepts handled requests
first server Express Nginx It's been started up until now 15 A connection
the second accepts Express Nginx It has been successfully created up to now 15 The second handshake
Third handled requests In total, we have dealt with 23 Requests
Number of requests lost = The number of handshakes - The number of connections , As you can see, no requests have been lost so far
Reading: 0 Writing: 1 Waiting: 1
Reading：Nginx Read... To the client Header Number of messages
Writing：Nginx Return to the client Header Number of messages
Waiting：Nginx Finished processing resident link waiting for next request instruction （ Turn on keep-alive Under the circumstances , This value is equal to Active - (Reading+Writing)

3.4.1 To configure Nginx Module

# Enable redis module cd /opt/metricbeat-7.10.0

./metricbeat modules enable nginx

# modify redis module To configure cd /opt/metricbeat-7.10.0/modules.d

vim nginx.yml

- module: nginx
#metricsets:
# - stubstatus
period: 10s
# Nginx hosts
#hosts: ["http://127.0.0.1"]
hosts: ["http://192.168.226.30"]
# Path to server status. Default server-status
#server_status_path: "server-status"
server_status_path: "nginx-status"
#username: "user"
#password: "secret"

# start-up cd /opt/metricbeat-7.10.0

./metricbeat -e

test ： request nginx, stay ES View in ：

You can see ,nginx The index data of has been written to Elasticsearch.

added Module Please refer to the official documents for use ：

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html

4. Kibana

https://www.elastic.co/cn/kibana

Kibana Is an open source data analysis and visualization platform , It is Elastic Stack One of the members , Designed for and Elasticsearch Collaboration . You can use Kibana Yes Elasticsearch Search the data in the index 、 see 、 interactions . You can easily use charts 、 Tables and maps provide diversified analysis and presentation of data .
4.1 configuration setup

# 1. download （ Pay attention to downloading and Elasticsearch Same version , Otherwise, it is incompatible and cannot be started ） Upload to server ,

https://www.elastic.co/cn/downloads/past-releases#kibana

decompression ： tar -zxvf kibana-7.9.3-linux-x86_64.tar.gz -C /opt/

# 2. Modify the configuration file cd /opt/ mv kibana-7.9.3-linux-x86_64/ kibana-7.9.3 cd kibana-7.9.3/

vim config/kibana.yml

server.host: "192.168.226.30" # The address of the exposure service

elasticsearch.hosts: ["http://192.168.226.30:9200","http://192.168.226.31:9200","http://192.168.226.32:9200"] # To configure Elasticsearch

more kibana To configure ：https://www.elastic.co/guide/en/kibana/7.9/settings.html

# 3. start-up

./bin/kibana Tips can't be used root start-up , Start with another account chown -R elsearch:elsearch kibana-7.9.3/ cd kibana-7.9.3/ su elsearch ./bin/kibana

There will be several warnings at startup ：

1. stay kibana.yml Add configuration at the end ：

xpack.security.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.reporting.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters" # arbitrarily 32 A string
xpack.reporting.capture.browser.chromium.disableSandbox: false # Turn off the sandbox

2. stay es Index .kibana_task_manager_1 Delete