Prometheus operation and maintenance tool promtool (III) debug function

Promtool  stay  debug  One aspect is  3  Sub orders , Used separately to obtain  profiling debug  Information 、 obtain  metric debug  Information 、 Get all  debug  Information 、 Unit test the rule file , Next, let's take a look at .

obtain  profiling debug  Information

Use   This command to get  Prometheus  Of  profiling  The performance data . The parameters of the command are as follows ：

promtool debug pprof <server>


Use this command to create a file. Take a look .

[[email protected] ~]# ./promtool debug pprof 'http://127.0.0.1:9090'
collecting: http://127.0.0.1:9090/debug/pprof/threadcreate
collecting: http://127.0.0.1:9090/debug/pprof/profile?seconds=30
collecting: http://127.0.0.1:9090/debug/pprof/block
collecting: http://127.0.0.1:9090/debug/pprof/goroutine
collecting: http://127.0.0.1:9090/debug/pprof/heap
collecting: http://127.0.0.1:9090/debug/pprof/mutex
collecting: http://127.0.0.1:9090/debug/pprof/trace?seconds=30
Compiling debug information complete, all files written in "debug.tar.gz".

[[email protected] ~]# tar -zxf debug.tar.gz
[[email protected] ~]# ll -h
-rw-rw-rw- 1 root root 177 1 month  1 1970 block.pb
-rw-rw-rw- 1 root root 5.9K 1 month  1 1970 cpu.pb
-rw-rw-rw- 1 root root 7.4K 1 month  1 1970 goroutine.pb
-rw-rw-rw- 1 root root 223K 1 month  1 1970 heap.pb
-rw-rw-rw- 1 root root 177 1 month  1 1970 mutex.pb
-rw-rw-rw- 1 root root 493 1 month  1 1970 threadcreate.pb
-rw-rw-rw- 1 root root 73K 1 month  1 1970 trace.pb


The created file name is  debug.tar.gz , Use  tar  After decompressing the command, you will get  block.pb 、cpu.pb、goroutine.pb、heap.pb、mutex.pb、threadcreate.pb、trace.pb  altogether  7  File .

These files can be used  pprof  Tools to analyze .

obtain  metric debug  Information

Promtool  Of  debug metrics  The subcommand is download  Prometheus  Provided  Metric  Index and compress . This command is seldom used , Because from  Prometheus  You can get these directly from the port of  Metric  data . The meaning of this debugging command is , When you can't solve the problem by yourself , You can put  Metric  Export is provided to other analysts for analysis and debugging , The parameter structure of this command is as follows ：

promtool debug metrics <server>


Let's use this command

[[email protected] ~]# ./promtool debug metrics 'http://127.0.0.1:9090'
collecting: http://127.0.0.1:9090/metrics
Compiling debug information complete, all files written in "debug.tar.gz".
[[email protected] ~]# tar -zxf debug.tar.gz
tar: metrics.txt： Untrusted old timestamp  1970-01-01 08:00:00
[[email protected] ~]# tail metrics.txt
# TYPE prometheus_web_federation_warnings_total counter
prometheus_web_federation_warnings_total 0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 519347
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0


You can see that this command generates a  debug.tar.gz  The compressed file of , After extracting this file, it is  metrics.txt , Cm is full  metric . Send this document to others , You can know the current  Prometheus  The state of .

This command has the same effect as the following command .

curl 127.0.0.1:9090/metrics > metrics.txt


Get all  debug  Information

Promtool  Of  debug all  Subcommand is to combine the results of the previous two subcommands , And then pack it , The usage of this subcommand is as follows ：

promtool debug all <server>


Try to execute ：

[[email protected] ~]# ./promtool debug all 'http://127.0.0.1:9090'
collecting: http://127.0.0.1:9090/debug/pprof/profile?seconds=30
collecting: http://127.0.0.1:9090/debug/pprof/block
collecting: http://127.0.0.1:9090/debug/pprof/goroutine
collecting: http://127.0.0.1:9090/debug/pprof/heap
collecting: http://127.0.0.1:9090/debug/pprof/mutex
collecting: http://127.0.0.1:9090/debug/pprof/threadcreate
collecting: http://127.0.0.1:9090/debug/pprof/trace?seconds=30
collecting: http://127.0.0.1:9090/metrics
Compiling debug information complete, all files written in "debug.tar.gz".

[[email protected] ~]# tar -zxf debug.tar.gz
tar: cpu.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: block.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: goroutine.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: heap.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: mutex.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: threadcreate.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: trace.pb： Untrusted old timestamp  1970-01-01 08:00:00
tar: metrics.txt： Untrusted old timestamp  1970-01-01 08:00:00
[[email protected] ~]# ls
block.pb debug.tar.gz heap.pb mutex.pb trace.pb
cpu.pb goroutine.pb metrics.txt threadcreate.pb


You can see that it is the same as the executive asking your family .

Test rule file

Promtool  You can also unit test the rule file , The command usage is as follows ：

promtool test rules <test-rule-file>...


Let's look for an alarm rule file to execute .

[[email protected] ~]# ./promtool test rules promtsdb.rules.yaml
Unit Testing: promtsdb.rules.yaml
 FAILED:
yaml: unmarshal errors:
 line 1: field groups not found in type main.unitTestFile


There's a misreport . At this time, we need to find the cause of the error . We met before  Promtool  Of  check rules  Sons command , These two commands are used to check the rule file , The difference is ,check rules  Only verify whether there are errors in the syntax of the file , and  test rules  Is to unit test the rule file .

summary

Today we checked  Promtool  Of  debug  part , Tomorrow we'll see  TSDB  part .

Prometheus  Operation and maintenance tools  Promtool （ One ）Check  function

Prometheus  Operation and maintenance tools  Promtool （ Two ）Query  function