当前位置：网站首页>Prometheus complete process of configuring alertmanager

Prometheus complete process of configuring alertmanager

2022-07-28 20:46:00 【51CTO】

prometheus Configuration of

1 stay prometheus Profile add alertmanager Address

       
       # Alertmanager configuration
       
alerting:
       
  alertmanagers:
       
    - static_configs:
       
        - targets:
       
          - xxx.xxx.xxx.xxx:9093
       
#xxx
      
1.
2.
3.
4.
5.
6.
7.

2 stay prometheus Add alarm rules to the configuration file

       
       #  Set the location of the rule file 
       
rule_files:
       
  - "rules/*.yml"
       
 #  Edit rule file 
       
vim rules/node_rules.yml 
       
groups:
       
- name: node-rules
       
  rules:
       
  - alert: node-up
       
    expr: up == 0
       
    for: 15s
       
    labels:
       
      severity: 1
       
      team: node
       
    annotations:
       
      summary: "{{$labels.instance}}Instance has been down for more than 5 minutes"
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

3 restart prometheus, Load the configuration

alertmanager Configuration of

1 alertmanager The configuration file

       
       vim alertmanager.yml
       
global:
       
  resolve_timeout: 5m
       
  smtp_smarthost: 'smtp.163.com:25'
       
  smtp_from: '[email protected]'
       
  smtp_auth_username: '[email protected]'
       
  smtp_auth_password: 'WJEZTKXVFDVVVV'
       
  smtp_require_tls: false
       
templates:
       
  - '/usr/local/alertmanager/email.tmpl'
       
route:
       
  group_by: ['alertname']
       
  group_wait: 30s
       
  group_interval: 5m
       
  repeat_interval: 1h
       
  receiver: 'email'
       
receivers:
       
  - name: 'email'
       
    email_configs:
       
      - to: '[email protected]'
       
        html: '{{ template "email.to.html" . }}'
       
        send_resolved: true
       
inhibit_rules:
       
  - source_match:
       
      severity: 'critical'
       
    target_match:
       
      severity: 'warning'
       
    equal: ['alertname']
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.

2 Edit the mail template file

       
       vim email.tmpl
       
{{ define "email.to.html" }}
       
{{ if gt (len .Alerts.Firing) 0 }}{{ range .Alerts }}
       
@ The alarm : <br>
       
 Alarm program : prometheus_alert <br>
       
 Alarm level : {{ .Labels.severity }}  level <br>
       
 Alarm type : {{ .Labels.alertname }} <br>
       
 Faulty host : {{ .Labels.instance }} <br>
       
 Warning topic : {{ .Annotations.summary }} <br>
       
 Alarm details : {{ .Annotations.description }} <br>
       
 Trigger time : {{ .StartsAt }} <br>
       
{{ end }}
       
{{ end }}
       
{{ if gt (len .Alerts.Resolved) 0 }}{{ range .Alerts }}
       
@ recovery : <br>
       
 Alarm host : {{ .Labels.instance }} <br>
       
 Warning topic : {{ .Annotations.summary }} <br>
       
 recovery time : {{ .EndsAt }} <br>
       
{{ end }}
       
{{ end }}
       
{{ end }}
       
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.

3 restart alertmanager

Examination result

stay prometheus see

prometheus To configure alertmanager The whole process _prometheus Configure alarms

Because in alertmanager In the configuration group_by Set up alertname, By tag name alertname And the value Grouping , So these three alarm messages should be in the same group , Send with an email

prometheus To configure alertmanager The whole process _prometheus Configure alarms _02

notes

alertmanager Three concepts

grouping Grouping According to the tag and tag value of the alarm rule Group the same alarms , Merge multiple alarms into one notification , Avoid sudden occurrence of a large number of alarm notifications

Inhibition Inhibition When An alarm has been sent , Stop sending this alarm repeatedly trigger Other abnormal or fault alarm information , for example mysql I hung up from the Library , Alarm notification has been sent , Then the master-slave synchronization alarm information of the slave library should not be sent repeatedly

silent Silences Provides a simple mechanism , Quickly silence the alarm according to the tag , Check the matching of the incoming alarm , If the received alarm conforms to the silent configuration ,Alertmanager No alarm notification will be sent , Silence needs to be in WEB UI The interface is set to temporarily mask the specified alarm notification

The main configuration section of the configuration file

global Global configuration

template Alarm information template , You can customize email、 Template of enterprise wechat

route Upon receipt of prometheus After the alarm of , Send the alarm information to the specified address ,Alertmanager Process the incoming alarm information , Match according to the defined rules and configuration . Routing can be understood as a tree structure , Set the first route Root node , Next is the child node , After each alarm comes in , It will match from top to bottom , When the node is matched, a notification will be sent .

receiver Receiver Each receiver needs to set a globally unique name , And set one or more notification methods ： mail 、 WeChat 、 Text messages, etc .

inhibit_rules Suppressor You can specify alarms that need to be ignored under specific conditions

Configuration file parameter comments

       
       vim alertmanager.yml
       
global:
       
  resolve_timeout: 5m
       
  smtp_smarthost: 'smtp.163.com:25'              #  Address of mailbox server 
       
  smtp_from: '[email protected]'                #  Mailbox used for alarm mail 
       
  smtp_auth_username: '[email protected]'       #  mailbox user 
       
  smtp_auth_password: 'WJEZTKXVFDVVVV'           #  Email login authorization code 
       
  smtp_require_tls: false
       
templates:
       
  - '/usr/local/alertmanager/email.tmpl'         #  Alarm mail template file 
       
route:
       
  group_by: ['alertname']             #  Label alertname grouping 
       
  group_wait: 30s                     #  In the same group ,30 The alarm appears within seconds , Send in a group 
       
  group_interval: 5m                  #  When sending a new alarm message , How long to wait 
       
  repeat_interval: 1h                 #  If an alarm message has been sent , Time interval for sending again 
       
  receiver: 'email'                   #  Specify the receiver for alarm messages 
       
receivers:
       
  - name: 'email'                     #  Set the name of the receiver 
       
    email_configs:
       
      - to: '[email protected]'        #  The address to receive the alarm email , Comma split 
       
        html: '{{ template "email.to.html" . }}'  #  Template used 
       
        send_resolved: true
       
inhibit_rules:                 #  Inhibition rules 
       
  - source_match:              #  When the source tag alarm starts , Suppress the alarm of the target tag 
       
      severity: 'critical'     
       
    target_match:
       
      severity: 'warning'
       
    equal: ['alertname', 'job', 'service']       #  These three tags must be matched in the alarm to be suppressed  
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.

Example of multi route matching ：

       
       route:
       
  group_wait: 10s
       
  group_interval: 30s
       
  repeat_interval: 30m
       
  receiver: "slack"
       
  routes:
       
    - receiver: "slack"
       
      group_wait: 10s
       
      match_re:
       
        severity: critical|warning
       
      continue: true
       
    - receiver: "pager"
       
      group_wait: 10s
       
      match_re:
       
        severity: critial
       
      continue: true
       
receivers:
       
  - name: "slack"
       
    slack_configs:
       
      - api_url: 'https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxx'
       
        send_resolved: true
       
        channel: 'monitoring'
       
        text: "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"
       
  - name: "pager"
       
    webhook_configs:
       
      - url: http://a.b.c.d:8080/send/sms
       
        send_resolved: true
      
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.