当前位置：网站首页>Prometheus monitors the correct posture of redis cluster

Prometheus monitors the correct posture of redis cluster

2022-07-05 01:52:00 【fsckzy】

Prometheus monitor Redis colony

Prometheus monitor Redis The right posture (redis colony )

Prometheus monitor Redis cluster, In fact, the routine is the same , Use exporter.

exporter Responsible for collecting indicators , adopt http Exposed to the Prometheus Pull .granafa Then display the data through these indicators .Prometheus The collected data will also determine whether to send to... According to the alarm rules you set Alertmanager, Alertmanager Then it is necessary to judge whether to send an alarm .

Alertmanager The alarm is divided into three stages

Inactive The rules that trigger the alarm will be sent here .
Pending The waiting time you set , That is, in the rules for
Firing Send alarm to email 、 Nails or something

Far away , Start monitoring Redis cluster

redis_exporter monitor Redis cluster

Monitor what applications , Use the corresponding exporter, You can find it on the official website . EXPORTERS AND INTEGRATIONS

Redis Use redis_exporter , link ： redis_exporter

Support Redis 2.x - 5.x

Installation and parameters

Download address

       
        wget https://github.com/oliver006/redis_exporter/releases/download/v1.3.5/redis_exporter-v1.3.5.linux-amd64.tar.gz   
        
tar zxvf redis_exporter-v1.3.5.linux-amd64.tar.gz
        
cd redis_exporter-v1.3.5.linux-amd64/
        
./redis_exporter <flags>
       
       
        1.
        2.
        3.
        4.

redis_exporter Many parameters are supported , There are only a few that are useful to us .

       
        ./redis_exporter --help
        
Usage of ./redis_exporter:
        
    -redis.addr string
        
        Address of the Redis instance to scrape (default "redis://localhost:6379")
        
    -redis.password string
        
        Password of the Redis instance to scrape
        
    -web.listen-address string
        
        Address to listen on for web interface and telemetry. (default ":9121")
       
       
        1.
        2.
        3.
        4.
        5.
        6.
        7.
        8.

Single instance redis monitor

       
        nohup ./redis_exporter -redis.addr 172.18.11.138:6379 -redis.password xxxxx &
       
       
        1.

Prometheus Add single instance

       
        - job_name: redis_since
        
    static_configs:
        
    - targets: ['172.18.11.138:9121']
       
       
        1.
        2.
        3.

Redis Cluster monitoring scheme

This is very laborious , I checked a lot of information on the Internet , Most of them monitor single instances

The scheme I tried ：

Both of the following will prompt authentication failure

       
        level=error msg="Redis INFO err: NOAUTH Authentication required."
       
       
        1.

Method 1

       
        nohup ./redis_exporter -redis.addr 172.18.11.139:7000 172.18.11.139:7001 172.18.11.140:7002 172.18.11.140:7003 172.18.11.141:7004 172.18.11.141:7005 -redis.password xxxxx &
       
       
        1.

Method 2

       
        nohup ./redis_exporter -redis.addr redis://h:Lcsmy.312==/@172.18.11.139:7000 redis://h:Lcsmy.312==/@172.18.11.139:7001 redis://h:Lcsmy.312==/@172.18.11.140:7002 redis://h:Lcsmy.312==/@172.18.11.140:7003 redis://h:Lcsmy.312==/@172.18.11.141:7004 redis://h:Lcsmy.312==/@172.18.11.141:7005 -redis.password xxxxx &
       
       
        1.

Originally wanted to take the most low Methods , One instance starts another redis_exporter. In this case , Many statements in the cluster can't be used , such as cluster_slot_fail. Abandon this method

       
        nohup ./redis_exporter -redis.addr 172.18.11.139:7000  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9121 > /dev/null 2>&1 &
        
nohup ./redis_exporter -redis.addr 172.18.11.139:7001  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9122 > /dev/null 2>&1 &
        
nohup ./redis_exporter -redis.addr 172.18.11.140:7002  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9123 > /dev/null 2>&1 &
        
nohup ./redis_exporter -redis.addr 172.18.11.140:7003  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9124 > /dev/null 2>&1 &
        
nohup ./redis_exporter -redis.addr 172.18.11.141:7004  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9125 > /dev/null 2>&1 &
        
nohup ./redis_exporter -redis.addr 172.18.11.141:7005  -redis.password xxxxxx  -web.listen-address 172.18.11.139:9126 > /dev/null 2>&1 &
       
       
        1.
        2.
        3.
        4.
        5.
        6.

Finally, I had to go github carry issue. Communicate with the author in my Chinglish , I finally understand ... In fact, the official documents have been written .

       
        scrape_configs:
        
  ## config for the multiple Redis targets that the exporter will scrape
        
  - job_name: 'redis_exporter_targets'
        
    static_configs:
        
      - targets:
        
        - redis://first-redis-host:6379
        
        - redis://second-redis-host:6379
        
        - redis://second-redis-host:6380
        
        - redis://second-redis-host:6381
        
    metrics_path: /scrape
        
    relabel_configs:
        
      - source_labels: [__address__]
        
        target_label: __param_target
        
      - source_labels: [__param_target]
        
        target_label: instance
        
      - target_label: __address__
        
        replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121
        

        
  ## config for scraping the exporter itself
        
  - job_name: 'redis_exporter'
        
    static_configs:
        
      - targets:
        
        - <<REDIS-EXPORTER-HOSTNAME>>:9121
       
       
        1.
        2.
        3.
        4.
        5.
        6.
        7.
        8.
        9.
        10.
        11.
        12.
        13.
        14.
        15.
        16.
        17.
        18.
        19.
        20.
        21.
        22.
        23.

Redis Cluster actual operation

start-up redis_exporter

       
        nohup ./redis_exporter -redis.password xxxxx  &
       
       
        1.

a key

stay prometheus How to configure it ：

       
        - job_name: 'redis_exporter_targets'
        
    static_configs:
        
      - targets:
        
        - redis://172.18.11.139:7000
        
        - redis://172.18.11.139:7001
        
        - redis://172.18.11.140:7002
        
        - redis://172.18.11.140:7003
        
        - redis://172.18.11.141:7004
        
        - redis://172.18.11.141:7005
        
    metrics_path: /scrape
        
    relabel_configs:
        
      - source_labels: [__address__]
        
        target_label: __param_target
        
      - source_labels: [__param_target]
        
        target_label: instance
        
      - target_label: __address__
        
        replacement: 172.18.11.139:9121
        
  ## config for scraping the exporter itself
        
  - job_name: 'redis_exporter'
        
    static_configs:
        
      - targets:
        
        - 172.18.11.139:9121
       
       
        1.
        2.
        3.
        4.
        5.
        6.
        7.
        8.
        9.
        10.
        11.
        12.
        13.
        14.
        15.
        16.
        17.
        18.
        19.
        20.
        21.
        22.

In this way, the data of the cluster can be collected . But the log suggests

       
        time="2019-12-17T09:10:49+08:00" level=error msg="Couldn't connect to redis instance"
       
       
        1.

I suddenly figured it out during my lunch break , As long as you can connect to a node in a cluster , Naturally, you can query the indicators of other nodes . So the startup command is changed to ：

       
        nohup ./redis_exporter -redis.addr 172.18.11.141:7005  -redis.password xxxxx &
       
       
        1.

Prometheus Configuration unchanged

Send some pictures ：

Prometheus monitor Redis The right posture of the group _redis

Prometheus monitor Redis The right posture of the group _Prometheus_02

Prometheus monitor Redis The right posture of the group _Prometheus_03

Alarm rules

       
        groups:
        
- name:  Redis
        
  rules: 
        
    - alert: RedisDown
        
      expr: redis_up  == 0
        
      for: 5m
        
      labels:
        
        severity: error
        
      annotations:
        
        summary: "Redis down (instance {{ $labels.instance }})"
        
        description: "Redis  Hang up ,mmp\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
        
    - alert: MissingBackup
        
      expr: time() - redis_rdb_last_save_timestamp_seconds > 60 * 60 * 24
        
      for: 5m
        
      labels:
        
        severity: error
        
      annotations:
        
        summary: "Missing backup (instance {{ $labels.instance }})"
        
        description: "Redis has not been backuped for 24 hours\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"       
        
    - alert: OutOfMemory
        
      expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 90
        
      for: 5m
        
      labels:
        
        severity: warning
        
      annotations:
        
        summary: "Out of memory (instance {{ $labels.instance }})"
        
        description: "Redis is running out of memory (> 90%)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
        
    - alert: ReplicationBroken
        
      expr: delta(redis_connected_slaves[1m]) < 0
        
      for: 5m
        
      labels:
        
        severity: error
        
      annotations:
        
        summary: "Replication broken (instance {{ $labels.instance }})"
        
        description: "Redis instance lost a slave\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
        
    - alert: TooManyConnections
        
      expr: redis_connected_clients > 1000
        
      for: 5m
        
      labels:
        
        severity: warning
        
      annotations:
        
        summary: "Too many connections (instance {{ $labels.instance }})"
        
        description: "Redis instance has too many connections\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"       
        
    - alert: NotEnoughConnections
        
      expr: redis_connected_clients < 5
        
      for: 5m
        
      labels:
        
        severity: warning
        
      annotations:
        
        summary: "Not enough connections (instance {{ $labels.instance }})"
        
        description: "Redis instance should have more connections (> 5)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
        
    - alert: RejectedConnections
        
      expr: increase(redis_rejected_connections_total[1m]) > 0
        
      for: 5m
        
      labels:
        
        severity: error
        
      annotations:
        
        summary: "Rejected connections (instance {{ $labels.instance }})"
        
        description: "Some connections to Redis has been rejected\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
       
       
        1.
        2.
        3.
        4.
        5.
        6.
        7.
        8.
        9.
        10.
        11.
        12.
        13.
        14.
        15.
        16.
        17.
        18.
        19.
        20.
        21.
        22.
        23.
        24.
        25.
        26.
        27.
        28.
        29.
        30.
        31.
        32.
        33.
        34.
        35.
        36.
        37.
        38.
        39.
        40.
        41.
        42.
        43.
        44.
        45.
        46.
        47.
        48.
        49.
        50.
        51.
        52.
        53.
        54.
        55.
        56.
        57.
        58.
        59.