当前位置:网站首页>Prometheus monitors the correct posture of redis cluster
Prometheus monitors the correct posture of redis cluster
2022-07-05 01:52:00 【fsckzy】
Prometheus monitor Redis colony
Prometheus monitor Redis The right posture (redis colony )
Prometheus
monitor Redis cluster
, In fact, the routine is the same , Use exporter
.
exporter
Responsible for collecting indicators , adopt http
Exposed to the Prometheus
Pull .granafa
Then display the data through these indicators .Prometheus
The collected data will also determine whether to send to... According to the alarm rules you set Alertmanager
, Alertmanager
Then it is necessary to judge whether to send an alarm .
Alertmanager
The alarm is divided into three stages
- Inactive The rules that trigger the alarm will be sent here .
- Pending The waiting time you set , That is, in the rules for
- Firing Send alarm to email 、 Nails or something
Far away , Start monitoring Redis cluster
redis_exporter monitor Redis cluster
Monitor what applications , Use the corresponding exporter
, You can find it on the official website . EXPORTERS AND INTEGRATIONS
Redis
Use redis_exporter
, link : redis_exporter
Support Redis 2.x - 5.x
Installation and parameters
wget https://github.com/oliver006/redis_exporter/releases/download/v1.3.5/redis_exporter-v1.3.5.linux-amd64.tar.gz
tar zxvf redis_exporter-v1.3.5.linux-amd64.tar.gz
cd redis_exporter-v1.3.5.linux-amd64/
./redis_exporter <flags>
- 1.
- 2.
- 3.
- 4.
redis_exporter
Many parameters are supported , There are only a few that are useful to us .
./redis_exporter --help
Usage of ./redis_exporter:
-redis.addr string
Address of the Redis instance to scrape (default "redis://localhost:6379")
-redis.password string
Password of the Redis instance to scrape
-web.listen-address string
Address to listen on for web interface and telemetry. (default ":9121")
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
Single instance redis monitor
nohup ./redis_exporter -redis.addr 172.18.11.138:6379 -redis.password xxxxx &
- 1.
Prometheus
Add single instance
- job_name: redis_since
static_configs:
- targets: ['172.18.11.138:9121']
- 1.
- 2.
- 3.
Redis Cluster monitoring scheme
This is very laborious , I checked a lot of information on the Internet , Most of them monitor single instances
The scheme I tried :
Both of the following will prompt authentication failure
level=error msg="Redis INFO err: NOAUTH Authentication required."
- 1.
Method 1
nohup ./redis_exporter -redis.addr 172.18.11.139:7000 172.18.11.139:7001 172.18.11.140:7002 172.18.11.140:7003 172.18.11.141:7004 172.18.11.141:7005 -redis.password xxxxx &
- 1.
Method 2
nohup ./redis_exporter -redis.addr redis://h:Lcsmy.312==/@172.18.11.139:7000 redis://h:Lcsmy.312==/@172.18.11.139:7001 redis://h:Lcsmy.312==/@172.18.11.140:7002 redis://h:Lcsmy.312==/@172.18.11.140:7003 redis://h:Lcsmy.312==/@172.18.11.141:7004 redis://h:Lcsmy.312==/@172.18.11.141:7005 -redis.password xxxxx &
- 1.
Originally wanted to take the most low Methods , One instance starts another redis_exporter
. In this case , Many statements in the cluster can't be used , such as cluster_slot_fail
. Abandon this method
nohup ./redis_exporter -redis.addr 172.18.11.139:7000 -redis.password xxxxxx -web.listen-address 172.18.11.139:9121 > /dev/null 2>&1 &
nohup ./redis_exporter -redis.addr 172.18.11.139:7001 -redis.password xxxxxx -web.listen-address 172.18.11.139:9122 > /dev/null 2>&1 &
nohup ./redis_exporter -redis.addr 172.18.11.140:7002 -redis.password xxxxxx -web.listen-address 172.18.11.139:9123 > /dev/null 2>&1 &
nohup ./redis_exporter -redis.addr 172.18.11.140:7003 -redis.password xxxxxx -web.listen-address 172.18.11.139:9124 > /dev/null 2>&1 &
nohup ./redis_exporter -redis.addr 172.18.11.141:7004 -redis.password xxxxxx -web.listen-address 172.18.11.139:9125 > /dev/null 2>&1 &
nohup ./redis_exporter -redis.addr 172.18.11.141:7005 -redis.password xxxxxx -web.listen-address 172.18.11.139:9126 > /dev/null 2>&1 &
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Finally, I had to go github
carry issue
. Communicate with the author in my Chinglish , I finally understand ... In fact, the official documents have been written .
scrape_configs:
## config for the multiple Redis targets that the exporter will scrape
- job_name: 'redis_exporter_targets'
static_configs:
- targets:
- redis://first-redis-host:6379
- redis://second-redis-host:6379
- redis://second-redis-host:6380
- redis://second-redis-host:6381
metrics_path: /scrape
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121
## config for scraping the exporter itself
- job_name: 'redis_exporter'
static_configs:
- targets:
- <<REDIS-EXPORTER-HOSTNAME>>:9121
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
Redis Cluster actual operation
start-up redis_exporter
nohup ./redis_exporter -redis.password xxxxx &
- 1.
a key
stay prometheus
How to configure it :
- job_name: 'redis_exporter_targets'
static_configs:
- targets:
- redis://172.18.11.139:7000
- redis://172.18.11.139:7001
- redis://172.18.11.140:7002
- redis://172.18.11.140:7003
- redis://172.18.11.141:7004
- redis://172.18.11.141:7005
metrics_path: /scrape
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 172.18.11.139:9121
## config for scraping the exporter itself
- job_name: 'redis_exporter'
static_configs:
- targets:
- 172.18.11.139:9121
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
In this way, the data of the cluster can be collected . But the log suggests
time="2019-12-17T09:10:49+08:00" level=error msg="Couldn't connect to redis instance"
- 1.
I suddenly figured it out during my lunch break , As long as you can connect to a node in a cluster , Naturally, you can query the indicators of other nodes . So the startup command is changed to :
nohup ./redis_exporter -redis.addr 172.18.11.141:7005 -redis.password xxxxx &
- 1.
Prometheus
Configuration unchanged
Send some pictures :
Alarm rules
groups:
- name: Redis
rules:
- alert: RedisDown
expr: redis_up == 0
for: 5m
labels:
severity: error
annotations:
summary: "Redis down (instance {{ $labels.instance }})"
description: "Redis Hang up ,mmp\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: MissingBackup
expr: time() - redis_rdb_last_save_timestamp_seconds > 60 * 60 * 24
for: 5m
labels:
severity: error
annotations:
summary: "Missing backup (instance {{ $labels.instance }})"
description: "Redis has not been backuped for 24 hours\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: OutOfMemory
expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "Out of memory (instance {{ $labels.instance }})"
description: "Redis is running out of memory (> 90%)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: ReplicationBroken
expr: delta(redis_connected_slaves[1m]) < 0
for: 5m
labels:
severity: error
annotations:
summary: "Replication broken (instance {{ $labels.instance }})"
description: "Redis instance lost a slave\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: TooManyConnections
expr: redis_connected_clients > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "Too many connections (instance {{ $labels.instance }})"
description: "Redis instance has too many connections\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: NotEnoughConnections
expr: redis_connected_clients < 5
for: 5m
labels:
severity: warning
annotations:
summary: "Not enough connections (instance {{ $labels.instance }})"
description: "Redis instance should have more connections (> 5)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: RejectedConnections
expr: increase(redis_rejected_connections_total[1m]) > 0
for: 5m
labels:
severity: error
annotations:
summary: "Rejected connections (instance {{ $labels.instance }})"
description: "Some connections to Redis has been rejected\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
边栏推荐
- 【LeetCode】88. Merge two ordered arrays
- How to build a technical team that will bring down the company?
- Kibana installation and configuration
- PHP Joseph Ring problem
- Database postragesq BSD authentication
- JVM - when multiple threads initialize the same class, only one thread is allowed to initialize
- Is there a sudden failure on the line? How to make emergency diagnosis, troubleshooting and recovery
- RichView TRVStyle MainRVStyle
- 19. Delete the penultimate node of the linked list
- [swagger]-swagger learning
猜你喜欢
Go RPC call
. Net starts again happy 20th birthday
PHP wechat official account development
Win:使用 PowerShell 检查无线信号的强弱
The perfect car for successful people: BMW X7! Superior performance, excellent comfort and safety
Logstash、Fluentd、Fluent Bit、Vector? How to choose the appropriate open source log collector
Blue Bridge Cup Square filling (DFS backtracking)
Remote control service
微信小程序:微群人脉微信小程序源码下载全新社群系统优化版支持代理会员系统功能超高收益
微信小程序:独立后台带分销功能月老办事处交友盲盒
随机推荐
Exploration and practice of integration of streaming and wholesale in jd.com
Practice of tdengine in TCL air conditioning energy management platform
小程序容器技术与物联网 IoT 可以碰撞出什么样的火花
Complex, complicated and numerous: illustration of seven types of code coupling
Restful fast request 2022.2.1 release, support curl import
The steering wheel can be turned for one and a half turns. Is there any difference between it and two turns
Incremental backup? db full
微信小程序:微群人脉微信小程序源码下载全新社群系统优化版支持代理会员系统功能超高收益
Some query constructors in laravel (2)
How to build a technical team that will bring down the company?
142. Circular linked list II
Win:将一般用户添加到 Local Admins 组中
官宣!第三届云原生编程挑战赛正式启动!
Armv8-a programming guide MMU (3)
MySQL REGEXP:正则表达式查询
PHP 约瑟夫环问题
batchnorm.py这个文件单GPU运行报错解决
Win:使用 Shadow Mode 查看远程用户的桌面会话
微信小程序:最新wordpress黑金壁纸微信小程序 二开修复版源码下载支持流量主收益
Phpstrom setting function annotation description