Prometheus adds email alarm and enterprise wechat robot alarm

We will be in  Prometheus  Write alert rules on the server  , These rules will use the metrics we collect and trigger alerts at specified thresholds or criteria , After receiving the alarm , Alertmanager  Alerts are processed and routed according to their tags . Once the path is determined , They will be made by Alertmanager Send to external destinations , E-mail 、 SMS or chat tool

Email alert （ Note that if you use qq Need to open smtp agreement ）

~]# vim alertmanager.yml 
global:
  resolve_timeout: 5m 
  smtp_smarthost: 'smtp.qq.com:465'        #  Mailbox url And port 
  smtp_from: '[email protected]'              #  Send email 
  smtp_auth_username: '[email protected]'    #  Send email 
  smtp_auth_password: 'xxxxxxxxxxxx'       #  Authorization code （ It can also be called password ）
  smtp_require_tls: false
  smtp_hello: 'qq.com'

templates:                                  #  Custom template                                                          
   - '/etc/alertmanager/template/*.tmpl'    

route:                                                                                    
  group_by: ['alertname','alarmPolicyName','version']                                     
  group_wait: 10s                                                                          
  group_interval: 10s                                                                      
  repeat_interval: 40m                                                                    
  receiver: 'email'

receivers:                                                                                
- name: 'email'                                                                           
  email_configs:                            #  Own mailbox                                               
  - to: '[email protected]'                                                                         
    html: '{
    { template "email.html" . }}'
    headers: { Subject: "[WARN]CPU The alarm " }
    send_resolved: true 

 

 

Enterprise wechat robot

1. Create the machine first

 

 

 2. establish webhook service

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: prometheus-webhook-dingtalk
  name: prometheus-webhook-dingtalk
  namespace: kube-system
spec:
  selector:
    matchLabels:
      run: prometheus-webhook-dingtalk
  template:
    metadata:
      labels:
        run: prometheus-webhook-dingtalk
    spec:
      containers:
      - args:
        - --adapter=/app/prometheusalert/wx.js=/adapter/wx=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key={}  #  there key Press the just created 
        image: registry.cn-hangzhou.aliyuncs.com/guyongquan/webhook-adapter  
        name: prometheus-webhook-dingtalk
        ports:
        - containerPort: 80
          protocol: TCP
 
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: prometheus-webhook-dingtalk
  name: prometheus-webhook-dingtalk
  namespace: kube-system
spec:
  ports:
  - port: 8060
    protocol: TCP
    targetPort: 80
  selector:
    run: prometheus-webhook-dingtalk
  type: ClusterIP

3. Alarm mode writing

~]# vim alertmanager.yaml
 
    global:
       resolve_timeout: 5m
 
    templates:
        - '/etc/alertmanager/template/*.tmpl'
 
    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 100m
      receiver: 'web.hook'
 
    receivers:
    - name: 'web.hook'
      webhook_configs:
      - url: 'http://prometheus-webhook-dingtalk.kube-system.svc.cluster.local:8060/adapter/wx'               #  Just created webhook Address 
        send_resolved: false
    
    inhibit_rules:
      - source_match:
           severity: 'critical'
        target_match:
           severity: 'warning'
        equal: ['alertname', 'dev', 'instance']

notes ： When configuring the alarm of enterprise wechat robot, you need to install webhook; The last restart

4. see