Prometheus + alertmanager message alert

install Prometheus

To configure prometheus.yml, The default configuration https://prometheus.io/docs/prometheus/latest/getting_started/

mkdir /opt/promethus
cd /opt/promethus/
vim prometheus.yml

Default prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

Pull the mirror image

docker pull prom/prometheus

Start installation

docker run -d \
    -p 9090:9090 \
    -v /opt/prometheus:/etc/prometheus \
    --name promethenus \
    --restart=always \
    prom/prometheus

-p Port mapping
-v Mount the file
-name Instance naming
–restart=always restart docker The container starts automatically

browser ip Add ports 9090 Can access

 

install AlertManager

To configure alertmanager.yml, The default configuration https://prometheus.io/docs/prometheus/latest/getting_started/

mkdir /opt/alertmanager
cd /opt/alertmanager/
vim alertmanager.yml

Default alertmanager.yml

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

browser ip Add ports 9090 Can access

Pull the mirror image

docker pull bitnami/alertmanager:latest

Start installation

docker run -d \
	-p 9093:9093 \
	--name alertmanager 
	-v /opt/alertmanager/alertmanager.yml:/opt/bitnami/alertmanager/conf/config.yml  \
	--restart=always \
	bitnami/alertmanager:latest

browser ip Add ports 9093 Can access

 

To configure Prometheus

prometheus.yml Example

global:
  scrape_interval:     15s
  external_labels:
    monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'

    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'service'
    static_configs:    
    - targets: ['192.168.xxx.0:9100']
    - targets: ['192.168.xxx.1:9100']

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - [ The server  ip]:9093
      
rule_files:
    - "rules.yml"

alerting.alertmanagers.static_configs.targets：AlertManager Deployed servers [ ip: port ]
rule_files： newly build rules.yml, Path and prometheus.yml In the same place

 
rules.yml Example

groups:
- name: CPU-rule
  rules:
  - alert: High-CPU-80
    expr: 100-avg(irate(node_cpu_seconds_total{
    job="component",mode="idle"}[5m]))by(instance)*100 > 80
    for: 1m
    labels:
      severity: warning 
    annotations:
      description: "{
    {$labels.instance}}: Client CPU is above 80% (current value is: {
    { $value }}"
  - alert: High-CPU-90
    expr: 100-avg(irate(node_cpu_seconds_total{
    job="component",mode="idle"}[5m]))by(instance)*100 > 90
    for: 1m
    labels:
      severity: warning 
    annotations:
      description: "{
    {$labels.instance}}: Client CPU is above 90% (current value is: {
    { $value }}"
      
- name: Menory-rule
  rules:
  - alert: HighMenory-80
    expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))* 100 > 80
    for: 1m
    labels:
      severity: warning 
    annotations: 
      description: "jobname:{
    {$labels.job}}, instance:{
    {$labels.instance}}, Client num is above 80%, current value is: {
    { $value }}" 
  - alert: HighMenory-90
    expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))* 100 > 90
    for: 1m
    labels:
      severity: warning 
    annotations: 
      description: "jobname:{
    {$labels.job}}, instance:{
    {$labels.instance}}, Client num is above 90%, current value is: {
    { $value }}"  
 
      
- name: jvm-rule
  rules:      
  - alert: High-jvm-80
    expr: jvm_memory_usage_after_gc_percent{
    } * 100 > 80
    for: 1m
    labels:
      severity: warning 
    annotations:
      description: "jobname:{
    {$labels.job}}, application :{
    {$labels.application }}, jvm num is above 80%, current value is: {
    { $value }}"        
  - alert: High-jvm-90
    expr: jvm_memory_usage_after_gc_percent{
    } * 100 > 90
    for: 1m
    labels:
      severity: warning 
    annotations:
      description: "jobname:{
    {$labels.job}}, application :{
    {$labels.application }}, jvm num is above 90%, current value is: {
    { $value }}"      

name： It's similar to grouping , The example is divided into cpu、 Memory 、jvm
alert： Custom naming
expr： Index value
for： The duration of the
label： label , Customize
annotations： annotation , Customize

Indicator Status
Monitor target status ,status - targets

 

To configure AlertManager

alertmanager.yml Example

global:
  smtp_smarthost: 'smtp.exmail.qq.com:465'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: '111'
  smtp_require_tls: false

route:
  receiver: mail
  
receivers:
- name: 'mail'
  email_configs:
  - to: '[email protected]'
  - to: '[email protected]'