当前位置：网站首页>Installation and configuration of grayog new generation log collection early warning system

Installation and configuration of grayog new generation log collection early warning system

2022-06-27 20:24:00 【51CTO】

In the company, I have been responsible for the construction of the monitoring system of the whole company , It mainly includes the hardware monitoring of the underlying server 、 System level monitoring 、 Monitoring of network equipment 、 Middleware and application level monitoring, etc . although Zabbix Support log monitoring , Because in the data volume 、 Search and log display are relatively weak , Only simple log alarms can be made . Therefore, log monitoring still needs to be done through professional tools . I compared it ELK、 Business log tool Splunk and Graylog, Later, I chose Graylog.Graylog Simply put, it is an open source log aggregation 、 analysis 、 Audit 、 Presentation and early warning tools . comparison ELK,Graylog A lightweight ,UI The interface is more beautiful , There are abundant and perfect API Interface . Currently, the logs of network devices are collected 、MySQL Error log 、Linux System logs, etc , We will consider accessing the microservice log later . Various logs are statistically analyzed according to the error level , By looking at Dashboards The report can confirm whether there are problems with thousands of online devices . The corresponding high-level logs pass Graylog +Python The program realizes the alarm function of wechat and e-mail .

One 、Graylog advantage

Zero development ： Collect from -> Storage -> analysis -> Present the complete process .
Deployment and maintenance are simple ： Integrated solutions , Unlike ELK Integration of three independent systems .
Multi day Zhiyuan access ：syslog、Filebeat、Log4j、Logstash etc. .
Multi protocol access ：UDP、TCP、HTTP、AMQP.
Custom panel ： Provide curves 、 The pie chart 、 A rich list of graphics such as world maps .
Full text search ： Support filtering and searching all logs by syntax .
Support alarm ： Log analysis platform with alarm function , Currently only email is supported , Can pass API Realize wechat alarm .
Rights management ： Flexible authority allocation and management .
Support clusters ： The platform performance can be extended according to the application .

Two 、Graylog Architecture design , Support clusters

GrayLog： Provide GrayLog External interface , Belong to CPU Concentrated .
Elasticsearch： Persistent storage and retrieval of log files , Belong to IO Concentrated .
MongoDB： Store some GrayLog Configuration information .

Graylog Installation and configuration of the new generation log collection and early warning system _mongodb

3、 ... and 、 Installation configuration

3.1 adopt Docker install mongodb database .

# obtain 3.6.5 Version of mongo Mirror image .
docker pull mongo:3.6.5

# Run container .
docker run -itd --name graylog-mongo01 -p 27017:27017 mongo:3.6.5 --auth 

# Get into mongo database , establish admin Users and grant superuser rights .
docker exec -it graylog-mongo01 mongo admin

> use admin
switched to db admin
> db.createUser(
...     {
...         user:"admin",
...         pwd:"xxxxxx",
...         roles:[{role:"root",db:"admin"}]
...     }
... );

# to graylog New user and password , You must switch the database to this database .
> use graylog
switched to db graylog

> db.createUser(
   {
      user:"grayloguser",
      pwd:"xxxxxx",
      roles:[{role:"readWrite",db:"graylog"}]
   }
);
# See if the user exists .
> show users;
{
        "_id" : "graylog.grayloguser",
        "user" : "grayloguser",
        "db" : "graylog",
        "roles" : [
                {
                        "role" : "readWrite",
                        "db" : "graylog"
                }
        ]
}

# Enter through user name and password mongo Inside the container 
[[email protected] ~]# docker exec -it graylog-mongo01 mongo admin -u admin -p
MongoDB shell version v3.6.5
Enter password:
connecting to: mongodb://127.0.0.1:27017/admin
MongoDB server version: 3.6.5


     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.
      22.
      23.
      24.
      25.
      26.
      27.
      28.
      29.
      30.
      31.
      32.
      33.
      34.
      35.
      36.
      37.
      38.
      39.
      40.
      41.
      42.
      43.
      44.
      45.
      46.
      47.
      48.
      49.
      50.
      51.

3.2 adopt docker compose Mode of installation ES.

[[email protected] ~]# cat /etc/docker/conf/docker-compose.yml 
version: '2.2'
services:
  sw-es01:
    image: xx.xx.xx.xx:5000/library/elasticsearch:7.6.0
    container_name: sw-es01
    restart: always
    environment:
      - cluster.name=sw-docker-cluster
      - node.name=sw-es01
      - discovery.type=single-node
      - network.host=0.0.0.0
      - network.publish_host=192.168.xx.xx
      - transport.tcp.port=9300
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - "ES_JAVA_OPTS=-Xms32g -Xmx32g"
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elsearch/data:/usr/share/elasticsearch/data
      - /data/elsearch/logs:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200


    network_mode: host


    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535

[[email protected] ~]# nohub docker-compose -f /etc/docker/conf/sw.yml up &

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.
      22.
      23.
      24.
      25.
      26.
      27.
      28.
      29.
      30.
      31.
      32.
      33.
      34.
      35.
      36.
      37.
      38.
      39.
      40.
      41.

3.3 install Graylog service .

establish graylog Profile directory .

install graylog service .

[[email protected] software]# tar -zxvf graylog-VERSION.tgz -C /usr/loca/
[[email protected] software ]# cd /usr/local/
[[email protected] local]# ln -s  graylog-VERSION graylog

     
      1.
      2.
      3.

Copy and edit graylog Configuration files and startup files .

[[email protected] local]# cp graylog/graylog.conf /etc/graylog/server/
[[email protected] local]# cp graylog/bin/graylogctl /etc/init.d/
[[email protected] ~]# grep '^[a-z]' /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/graylog-node-1
password_secret = Z2LoxxeFvWoAPbMF0sIlYWhHH06leW6bfUAeImqhUe86Wzq8p4HDZAyKTQpaedvBuCoKYjaQAGQTj93R33sREiSIVt1sTRg0
root_username = admin
root_password_sha2 = xxxxxx
root_timezone = Asia/Shanghai
bin_dir = bin
data_dir = data
plugin_dir = plugin
http_bind_address = 0.0.0.0:8080
http_publish_uri = http://192.168.xx.xx:8080/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
# Appoint mongodb Database user name and password .
mongodb_uri = mongodb://grayloguser:[email protected]:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
# Appoint graylog The mail service , Post alarm can be sent by email .
transport_email_enabled = true
transport_email_hostname = 192.168.xx.xx
transport_email_port = 25
transport_email_use_auth = false
transport_email_subject_prefix = [graylog]
transport_email_from_email = [email protected]
proxied_requests_thread_pool_size = 32

     
      1.
      2.
      3.
      4.
      5.
      6.
      7.
      8.
      9.
      10.
      11.
      12.
      13.
      14.
      15.
      16.
      17.
      18.
      19.
      20.
      21.
      22.
      23.
      24.
      25.
      26.
      27.
      28.
      29.
      30.
      31.
      32.
      33.
      34.
      35.
      36.
      37.
      38.
      39.
      40.
      41.
      42.
      43.
      44.
      45.
      46.
      47.
      48.
      49.
      50.

start-up graylog service .

3.4 verification graylog Whether the service installation configuration is normal . adopt http://yourip:9000 To visit .

Graylog Installation and configuration of the new generation log collection and early warning system _elasticsearch_02

Four 、 summary

1）root_password_sha2 The password can be echo -n yourpassword | sha256sum To generate , there yourpassword Is used to log in graylog Of admin User's password . Generated root_password_sha2 Please delete the blank space and -, Otherwise, the login will not succeed .

2） edit graylog Startup file , Appoint graylog The path to the configuration file .

原网站

版权声明
本文为[51CTO]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/178/202206271817466996.html