Prometheus alarm process and related time parameter description

explain

use prometheus Do monitoring , There are many processes from the occurrence of an alarm event to the receipt of an alarm message , Understand the process and related time configuration , Can be more timely 、 Get alarm information efficiently .

Record below prometheus Alarm life cycle / technological process 、 Description of relevant configuration parameters and alarm cases .

prometheus Alarm life cycle / technological process

1. prometheus Regularly collect index data
2. prometheus Regularly calculate whether the indicator triggers rules
3. The indicator alarm status of the triggering rule changes to pending, When the duration exceeds for After the specified time , Convert to firing, And send the alarm to alertmanager
4. alertmanager After receiving the alarm , Wait for a grouping time , Send an alarm after the time ; If the packet continues to receive an alarm , Wait for a group alarm interval , Send an alarm for the packet again
5. If the alarm persists ,alertmanager The alarm will be sent repeatedly according to the retransmission interval

The following picture shows the whole prometheus Process panorama of , Can clearly understand prometheus Alarm operation process of .

Time related parameters

Parameter name	explain	The default value is	Parameter
scrape_interval	Index data collection interval	1 minute	prometheus.yml
evaluation_interval	Regular calculation interval	1 minute	prometheus.yml
for: Time	How long does the abnormality last to send an alarm	0	Rule configuration
group_wait	Group wait time . How long to wait for the first alarm to be sent in the same packet , The purpose is to send the same group of messages at the same time	30 second	alertmanager.yml
group_interval	The interval between the upper and lower groups sending alarms . Wait after the first alarm group_interval Time , Start to trigger a new alarm for this group	5 minute	alertmanager.yml
repeat_interval	Retransmission interval . The alarm has been sent , And there is no new alarm , The interval required to send the alarm again	4 Hours	alertmanager.yml

Case study

monitor Kafka Is the node down fall .

To configure

 Index name ：kakfa_up_status
1 Survive   0 Hang up 

# prometheus.yml To configure 
global:
  scrape_interval:     20s
  evaluation_interval: 30s

#  Rule configuration 
  - alert: kakfa_down
    expr: kakfa_up_status == 0
    for: 1m
    annotations:
      summary: "Kafka Hang up "

# alertmanager To configure 
route:
  group_by: [alertname]
  group_wait: 60s
  group_interval: 5m
  repeat_interval: 10m

Event flow

10:00:05 Kafka Hang up 
10:00:20  Pull indicators kakfa_up_status=0
10:00:30  Calculation rules , Find out Kafka Hang up , take kakfa_down Set to pending
10:00:30~10:01:30  Continuously pull indicators 、 Calculation rules 
10:01:30 kafka_down The duration reached 1 minute , Set to firing, Send to alertmanager
10:01:30 alertmanager After receipt of , Wait group wait time 
10:02:30  Group wait time complete , Give an alarm 
10:12:30  The alarm has not been resolved , Repeat the alarm 

Reference resources

prometheus Alarm mechanism -（ Why is the alarm not sent in time ） https://blog.csdn.net/luo4105/article/details/123700003

How soon can I receive prometheus Alarm of ？ https://www.jianshu.com/p/b3b4e68409e0

prometheus The alarm group_wait&repeat_interval https://blog.csdn.net/tryyourbest0928/article/details/115337984