Prometheus operation and maintenance tool promtool (IV) TSDB function

Promtool  stay  TSDB  One aspect is  6  Sub orders , Used for writing performance test respectively 、 analysis 、 List the blocks 、dump、 from  OpenMetric  Import data block 、 Create data blocks for new record rules , Next, let's take a look at .

This is probably the first article written  Promtool TSDB  The article on . No relevant information was found on the Internet .

Write performance test

Promtool  It can be done to  Prometheus  Write performance test , The command parameters are as follows ：

[[email protected] ~]# ./promtool tsdb bench write --help
usage: promtool tsdb bench write [<flags>] [<file>]

Run a write performance benchmark.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
 details.
 --out=&quot;benchout&quot; Set the output path.
 --metrics=10000 Number of metrics to read.
 --scrapes=3000 Number of scrapes to simulate.

Args:
 [<file>] Input file with samples data, default is (../../tsdb/testdata/20kseries.json).


First, download the data needed for the test from the official website  20kseries.json  file , This file is in the source code warehouse  /tsdb/testdata/  Next , You can download through this command

wget https://raw.githubusercontent.com/prometheus/prometheus/main/tsdb/testdata/20kseries.json


After downloading , Appoint  metrics  and  scrapes  Two parameters are tested

[[email protected] ~]# ./promtool tsdb bench write --metrics=10000 --scrapes=3000 ./20kseries.json
level=info ts=2022-07-27T13:21:25.546626055Z caller=head.go:493 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-07-27T13:21:25.546734166Z caller=head.go:536 msg="On-disk memory mappable chunks replay completed" duration=11.815µs
level=info ts=2022-07-27T13:21:25.546766084Z caller=head.go:542 msg="Replaying WAL, this may take a while"
level=info ts=2022-07-27T13:21:25.547101874Z caller=head.go:613 msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2022-07-27T13:21:25.547131383Z caller=head.go:619 msg="WAL replay completed" checkpoint_replay_duration=38.26µs wal_replay_duration=315.491µs total_replay_duration=409.177µs
level=info ts=2022-07-27T13:21:25.549132675Z caller=db.go:1467 msg="Compactions disabled"
>> start stage=readData
>> completed stage=readData duration=145.973395ms
>> start stage=ingestScrapes
ingestion completed
>> completed stage=ingestScrapes duration=3.628682202s
 > total samples: 30000000
 > samples/sec: 8.267435217071414e+06
>> start stage=stopStorage
>> completed stage=stopStorage duration=1.522008202s


This test will generate a  benchout  Folder , Inside is the file generated by the test . This path can also be modified through parameters , Generate to other places or other names .

Simple view , Reading data this time consumed  145ms , Storing data consumes  1.5s .

Interpretation of this result , We are going to make a special explanation later .

TSDB  analysis

Promtool  It can be done to  Prometheus  Data block analysis , analysis  churn 、label  For technology and compression efficiency , The command parameters are as follows ：

[[email protected] ~]# ./promtool tsdb analyze --help
usage: promtool tsdb analyze [<flags>] [<db path>] [<block id>]

Analyze churn, label pair cardinality and compaction efficiency.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
 details.
 --limit=20 How many items to show in each list.
 --extended Run extended analysis.

Args:
 [<db path>] Database path (default is data/).
 [<block id>] Block to analyze (default is the last block).


When analyzing, you can specify the maximum number of items displayed in each list , Data must be specified  data  Catalog ,block id  It can be specified or not , If not specified, the latest one will be analyzed  block . After executing the command, you can see the following output .

[[email protected] ~]# ./promtool tsdb analyze ./prometheus/data/
Block ID: 01G8ZXKAHRGF7BV0FTQYZ18FH5
Duration: 59m55.713s
Series: 606677
Label names: 26
Postings (unique label pairs): 13190
Postings entries (total label pairs): 7819776

Label pairs most involved in churning:
329487 workload_user_cattle_io_workloadselector=deployment-appops-pushgateway-flink
329487 pod_template_hash=6d695f75cf
329487 kubernetes_namespace=appops
62797 kubernetes_pod_name=pushgateway-flink-6d695f75cf-2xhfh
58524 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8fq58
52495 kubernetes_pod_name=pushgateway-flink-6d695f75cf-d76n7
51544 kubernetes_pod_name=pushgateway-flink-6d695f75cf-z4bjb
22074 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9jgdk
21516 kubernetes_pod_name=pushgateway-flink-6d695f75cf-96hpq
20426 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9twr2
20047 kubernetes_pod_name=pushgateway-flink-6d695f75cf-ngzwz
11819 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8hv9z
8242 kubernetes_pod_name=pushgateway-flink-6d695f75cf-pddrc
3783 __name__=flink_taskmanager_Status_JVM_Memory_Heap_Used
3783 __name__=flink_taskmanager_job_task_checkpointAlignmentTime
3783 __name__=flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
3783 __name__=flink_taskmanager_Status_JVM_Threads_Count
3783 __name__=flink_taskmanager_Status_Shuffle_Netty_UsedMemorySegments
3783 __name__=flink_taskmanager_Status_Flink_Memory_Managed_Used
3783 __name__=flink_taskmanager_job_task_Shuffle_Netty_Output_Buffers_outputQueueLength

Label names most involved in churning:
329487 kubernetes_namespace
329487 kubernetes_pod_name
329487 workload_user_cattle_io_workloadselector
329487 __name__
329487 pod_template_hash
329487 job
314352 instance
314050 host
314050 tm_id
185403 task_attempt_num
185403 task_name
185403 subtask_index
185403 job_name
185403 task_id
185403 task_attempt_id
185403 job_id
15134 operator_name
15134 operator_id
66 method
55 quantile

Most common label pairs:
606565 kubernetes_namespace=appops
606565 workload_user_cattle_io_workloadselector=deployment-pushgateway-flink
606565 pod_template_hash=6d695f75cf
87063 kubernetes_pod_name=pushgateway-flink-6d695f75cf-ngzwz
87063 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9jgdk
87062 kubernetes_pod_name=pushgateway-flink-6d695f75cf-96hpq
87062 kubernetes_pod_name=pushgateway-flink-6d695f75cf-9twr2
67920 kubernetes_pod_name=pushgateway-flink-6d695f75cf-2xhfh
62700 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8fq58
54174 kubernetes_pod_name=pushgateway-flink-6d695f75cf-d76n7
53304 kubernetes_pod_name=pushgateway-flink-6d695f75cf-z4bjb
11892 kubernetes_pod_name=pushgateway-flink-6d695f75cf-8hv9z
8325 kubernetes_pod_name=pushgateway-flink-6d695f75cf-pddrc
6965 __name__=flink_taskmanager_job_task_Shuffle_Netty_Input_Buffers_inputFloatingBuffersUsage
6965 __name__=flink_taskmanager_job_task_numBuffersOut
6965 __name__=flink_taskmanager_job_task_isBackPressured
6965 __name__=flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
6965 __name__=flink_taskmanager_job_task_numBuffersOutPerSecond
6965 __name__=push_time_seconds
6965 __name__=flink_taskmanager_job_task_buffers_inPoolUsage

Label names with highest cumulative label value length:
50890 task_name
42890 tm_id
34890 operator_id
34890 task_id
34890 task_attempt_id
34890 job_id
21890 job_name
18928 job
18890 operator_name
13890 host
5972 __name__
3890 task_attempt_num
3890 subtask_index
3100 instance
340 kubernetes_pod_name
40 revision
38 handler
35 workload_user_cattle_io_workloadselector
19 quantile
16 method

Highest cardinality labels:
1012 instance
1002 job
1000 operator_id
1000 task_id
1000 task_attempt_num
1000 host
1000 task_name
1000 tm_id
1000 job_id
1000 job_name
1000 task_attempt_id
1000 subtask_index
1000 operator_name
137 __name__
10 kubernetes_pod_name
7 handler
7 quantile
4 method
3 code
2 version

Highest cardinality metric names:
6965 my_batch_job_duration_seconds
6965 flink_taskmanager_Status_Flink_Memory_Managed_Used
6965 flink_taskmanager_Status_JVM_CPU_Load
6965 flink_taskmanager_Status_JVM_CPU_Time
6965 flink_taskmanager_Status_JVM_ClassLoader_ClassesLoaded
6965 flink_taskmanager_Status_JVM_ClassLoader_ClassesUnloaded
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_MarkSweep_Count
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_MarkSweep_Time
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_Scavenge_Count
6965 flink_taskmanager_Status_JVM_GarbageCollector_PS_Scavenge_Time
6965 flink_taskmanager_Status_JVM_Memory_Direct_Count
6965 flink_taskmanager_Status_JVM_Memory_Direct_MemoryUsed
6965 flink_taskmanager_Status_JVM_Memory_Direct_TotalCapacity
6965 flink_taskmanager_Status_JVM_Memory_Heap_Committed
6965 flink_taskmanager_Status_JVM_Memory_Heap_Max
6965 flink_taskmanager_Status_JVM_Memory_Heap_Used
6965 flink_taskmanager_Status_JVM_Memory_Mapped_Count
6965 flink_taskmanager_Status_JVM_Memory_Mapped_MemoryUsed
6965 flink_taskmanager_Status_JVM_Memory_Mapped_TotalCapacity
6965 flink_taskmanager_Status_JVM_Memory_Metaspace_Committed


List  TSDB  Data blocks

Use  Promtool  You can also list the current  Prometheus  All blocks of data . The parameters of the command are as follows ：

[[email protected] ~]# ./promtool tsdb list --help
usage: promtool tsdb list [<flags>] [<db path>]

List tsdb blocks.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
 details.
 -r, --human-readable Print human readable values.

Args:
 [<db path>] Database path (default is data/).


Execute this command and let's see the effect , Advice and  

-r

  Parameters

[[email protected] ~]# ./promtool tsdb list -r ./prometheus-pushgateway/data/
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01G8XB6J42S6FTDMAPYV30CTR3 2022-07-26 12:00:03 +0000 UTC 2022-07-26 13:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 92MiB217KiB450B
01G8XEMDR374M816NZ2NMNS5AQ 2022-07-26 13:00:03 +0000 UTC 2022-07-26 14:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 92MiB217KiB825B
01G8XJ29C2JVV8PEQFR748C1QR 2022-07-26 14:00:03 +0000 UTC 2022-07-26 15:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 94MiB125KiB158B
01G8XNG502K46CF1960R1FYEAA 2022-07-26 15:00:03 +0000 UTC 2022-07-26 16:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 92MiB388KiB448B
01G8XRY0M2CV8J323VWM393JTZ 2022-07-26 16:00:03 +0000 UTC 2022-07-26 17:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 93MiB978KiB543B
01G8XWBW82NGHBPP6QPZMJ6XAM 2022-07-26 17:00:03 +0000 UTC 2022-07-26 18:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 93MiB979KiB590B
01G8XZSQW2WY7SG70Z40CW2CZV 2022-07-26 18:00:03 +0000 UTC 2022-07-26 19:00:00 +0000 UTC 59m56.018s 78421200 435860 435412 94MiB719KiB630B


Through this command, we can see  Block ID、 Start time and end time of data storage , How long has it been stored   data ,Sample  The number of ,Chunk  The number of ,Serie  The number of 、 as well as  Block  Size .

Dump

Promtool  OK, it's OK  dump, Just from  TSDB  in  dump Sample  Data out .

[[email protected] ~]# ./promtool tsdb dump --help
usage: promtool tsdb dump [<flags>] [<db path>]

Dump samples from a TSDB.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
 details.
 --min-time=-9223372036854775808
 Minimum timestamp to dump.
 --max-time=9223372036854775807
 Maximum timestamp to dump.

Args:
 [<db path>] Database path (default is data/).


For this command, be sure to specify the data directory  data  The path of , In addition, specify the maximum and minimum time , Otherwise  dump  All of the... In the database  Sample  data .

Let's execute this order

[[email protected] ~]# ./promtool tsdb dump --min-time=1658927217000 --max-time=1658930818000 ../prometheus-pushgateway/data/ 


This command  dump  It seems that the data will only be output on the screen , It will not be output to file ,dump  Remember to import to a file manually .

from  OpenMetric  Import data block

Promtool  Tools can also be from  OpenMetrics  Import import  sample  Data and generate  TSDB  Data blocks . The usage scenario of this command is to migrate data between different monitoring systems or time series databases  , First convert the corresponding data into  OpenMetric  Format , And then  OpenMetric  Format data into  Prometheus  Of  TSDB  In the database .

The parameters of this command are as follows :

[[email protected] ~]# ./promtool tsdb create-blocks-from openmetrics --help
usage: promtool tsdb create-blocks-from openmetrics <input file> [<output directory>]

Import samples from OpenMetrics input and produce TSDB blocks. Please refer to the storage docs for more details.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and more
 details.
 -r, --human-readable Print human readable values.
 -q, --quiet Do not print created blocks.

Args:
 <input file> OpenMetrics file to read samples from.
 [<output directory>] Output directory for generated blocks.


I have no corresponding experimental environment , Parameter tips let you refer to the storage documentation for more details , This is the document  https://prometheus.io/docs/prometheus/latest/storage/ . Let's learn by ourselves , I have an environment and am writing an article .

Create data blocks for new record rules

When a new record rule is created , This new recording rule has no historical data . The data generated by the recording rule is only stored in the database when it is created ,Promtool  You can create historical data of recording rules .

[[email protected] ~]# ./promtool tsdb create-blocks-from rules --help
usage: promtool tsdb create-blocks-from rules --start=START [<flags>] <rule-files>...

Create blocks of data for new recording rules.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/ for the options and
 more details.
 -r, --human-readable Print human readable values.
 -q, --quiet Do not print created blocks.
 --url=http://localhost:9090
 The URL for the Prometheus API with the data where the rule will be backfilled
 from.
 --start=START The time to start backfilling the new rule from. Must be a RFC3339 formatted date
 or Unix timestamp. Required.
 --end=END If an end time is provided, all recording rules in the rule files provided will
 be backfilled to the end time. Default will backfill up to 3 hours ago. Must be a
 RFC3339 formatted date or Unix timestamp.
 --output-dir=&quot;data/&quot; Output directory for generated blocks.
 --eval-interval=60s How frequently to evaluate rules when backfilling if a value is not set in the
 recording rule files.

Args:
 <rule-files> A list of one or more files containing recording rules to be backfilled. All recording rules
 listed in the files will be backfilled. Alerting rules are not evaluated.


Let's take a look at this order , The record rule file provided in the command should be a common  Prometheus  Rules file , Then specify the time range of the historical data to be generated ,

[[email protected] ~]# ./promtool tsdb create-blocks-from rules \
 --start 1617079873 \
 --end 1617097873 \
 --url http://mypromserver.com:9090 \
 rules.yaml rules2.yaml


command  

promtool tsdb create-blocks-from rules

  The output result of is a directory , This directory contains historical rule data blocks that record all rules in the rule file . By default , Output directory is data/. In order to take advantage of these new block data , These blocks must be moved to the running  Prometheus  Instance data directory , And then in  Prometheus  When the instance starts, set the parameter  

 --storage.tsdb.allow-overlapping-blocks

  Set to enable . after  Prometheus  At the next compression run , Merge the new block with the existing block . In this way, the historical data generated by the rule file can be recorded in  Prometheus  Query in .

There are some restrictions on creating historical data for recording rules ,

• If you execute multiple times to generate historical data  , And the start time and end time overlap , Then each time the record rule is run to generate historical data, a block containing the same data will be created .

• All rules in the record rules file will be evaluated .

• If the time interval is set in the record rule file , This interval takes precedence over  tsdb create-blocks-from rules  In the command  
  --eval-interval
    Parameters .

• If the alarm rule is in the record rule file , Then the alarm rule will be ignored .

• Rules in the same group cannot see the results of previous rules . This means that rules that reference other rules that are generating historical data are not supported . One solution is to generate historical data many times and create dependent data first ( And move dependent data to  Prometheus  Server data directory , So that  Prometheus API  Visit it ).

summary

4  A small series of chapter has been completed . The first  4  In the process of writing , In addition to official documents , There is no other reference , I hope there are no mistakes .