Log management platform of infrastructure and nail & email alarm notification

In the modern software development process , Logs are very important , It's impossible to be scattered in various projects , Wait until you check the log The server Go to a specific directory to check , It's obviously cumbersome and inefficient , All integrate a set of log management platform , It's also very important , This article is the first to build a log management platform , The second part is alarm and email notification

Connected to a , We continue to explain how to put ELK Combined with nailing and email sending functions , Let's know the important logs in time and give feedback quickly .

1. Sentinel install , The project is introduced in https://github.com/sirensolutions/sentinl, Click to introduce in detail .

Run the command to kibana Of bin Catalog

cd /usr/share/kibana/bin

Copy

, This is the default installation path , Then execute the following command

 ./kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.3.2-beta-1/sentinl-v6.3.1.zip

Copy

, Here, according to the server performance and network environment , It may take a while to see the success message , It will refresh automatically after success kibana service , Open again kibana, As shown in the figure, the installation is successful

1. Mail notification

a) To send an email , You have to have one first SMTP Send service , What I use here is 163, Now almost all service providers that provide email function can start SMTP function , Just open it .

b) Email configuration , open /etc/kibana/kibana.yml file , Add the following settings

sentinl:

  settings:

    email:

      active: true

      user: [email protected]

      password: 123456

      host: smtp.163.com

      ssl: false   # Add... According to the actual situation 

    report:

            active: true

Copy

Be sure to set the level , Otherwise, there will be inexplicable mistakes .

c) Click on sentinl, Add one Watcher, The configuration information here is as follows .

{

  "actions": {

    "Alerm": {

      "throttle_period": "1h0m0s",

      "email": {

        "to": "[email protected]",

        "from": "[email protected]",

        "subject": " Business system alarm ",

        "priority": "high",

        "html": " System program error alarm :  A total of {{payload.hits.total}}  Time , Please login to check <a herf=\"http://192.168.1.215:5601\" target=\"_blank\"> Click login </a>"

      }

    }

  },

  "input": {

    "search": {

      "request": {

        "index": [

          "nlog*"

        ],

        "body": {

          "query": {

            "bool": {

              "must": [

                {

                  "query_string": {

                    "analyze_wildcard": true,

                    "query": "\"error\""

                  }

                },

                {

                  "range": {

                    "@timestamp": {

                      "gte": "now-1h",

                      "lte": "now",

                      "format": "epoch_millis"

                    }

                  }

                }

              ],

              "must_not": []

            }

          }

        }

      }

    }

  },

  "condition": {

    "script": {

      "script": "payload.hits.total >= 5"

    }

  },

  "trigger": {

    "schedule": {

      "later": "every 2 minutes"

    }

  },

  "disable": true,

  "report": false,

  "title": "nlog",

  "wizard": {},

  "save_payload": false,

  "spy": false,

  "impersonate": false

}

Copy

It mainly configures the recipient mailbox , The basis of judgment , Judge the condition , Trigger interval

d) Turn on Watcher And implement , As shown in the figure below .

wait a moment , Will be in alarms See the alarm information under .

Open the mail of the recipient in the configuration file , You can see the information you receive , My message is as follows

, It indicates that the email alarm function operates normally .

　　2. Pin message

　　　　a) Pin receive message , It mainly uses the robot function of nail , First, open the robot , My robot settings are as follows

　　 Mainly to make use of webhook, Next, we use .

b) nailing Watcher Set up , Set as follows

{
  "actions": {
    "Webhook_683bd385-86b3-46ba-8e1b-f89cccccbbec": {
      "name": "WatcherName",
      "throttle_period": "1m",
      "webhook": {
        "priority": "high",
        "stateless": false,
        "method": "POST",
        "host": "oapi.dingtalk.com",
        "port": "443",
        "path": "/robot/send?access_token=ec5fe24b4a218f71bca667975850cbf3f2830b9bd2bd91f60ca07fab28a3d439",
        "body": " {\"msgtype\": \"text\",\r\n    \"text\": {\r\n        \"content\":\" Business system alarm \"\r\n          }\r\n}",
        "params": {
          "watcher": "{{watcher.title}}",
          "payload_count": "{{payload.hits.total}}"
        },
        "headers": {
          "Content-Type": "application/json"
        },
        "auth": " Nailing account name : password ",
        "message": " Business function alarm ",
        "use_https": true
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "index": [
          "nlog*"
        ],
        "body": {
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-15m/m",
                    "lte": "now/m",
                    "format": "epoch_millis"
                  }
                }
              }
            }
          },
          "size": 0,
          "aggs": {
            "dateAgg": {
              "date_histogram": {
                "field": "@timestamp",
                "time_zone": "Europe/Amsterdam",
                "interval": "1m",
                "min_doc_count": 1
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "script": "payload.aggregations.dateAgg.buckets.some(b => b.doc_count>=5)"
    }
  },
  "trigger": {
    "schedule": {
      "later": "every 1 minutes"
    }
  },
  "disable": true,
  "report": false,
  "title": "nlog_dingding",
  "wizard": {},
  "save_payload": false,
  "spy": false,
  "impersonate": false
}

Copy

c) Start and execute Watcher, Pay attention to the nail , If you receive the following information , It indicates that the function is running successfully .

such ,ELK Connect with the business system , And mail 、 Nail alarm has been introduced , In fact, give play to imagination and according to business needs , It can extend more useful functions . If you have better suggestions, welcome to communicate and make common progress .

Remember to pay attention to your favorite friends ~