A shell script to realize automatic expiration of Prometheus push gateway indicators

Prometheus Push Gateway Officials have said that they will not automatically expire the target . If there is a need , Here's a simple Shell Script , Delete all over 60 The indicator that is not pushed in seconds ：

baseurl=localhost:9091
for uri in $(curl -sS $baseurl/api/v1/metrics | jq -r '
  .data[].push_time_seconds.metrics[0] |
  select((now - (.value | tonumber)) > 60) |
  (.labels as $labels | ["job", "instance"] | map(.+"/"+$labels[.]) | join("/"))
'); do
  curl -XDELETE $baseurl/metrics/$uri | exit
  echo curl -XDELETE $baseurl/metrics/$uri
done

The prerequisite for running the above script is to install jq and curl .

explain ：

1. Push Gateway Press Group To manage metrics , Every Group Contains a set of indicators and a unique Group Labels.
2. Push Gateway Of API /api/vi/metrics You can get all Group And the current index value 、 Last push time .
3. Use jq Languages are filtered by last push time Group, And then use DELETE API Delete the group .
4. Group Labels yes Group Unique identification of , Depending on Push Client, Every Group There may be different Labels. Generally, there are... By default job and instance, If there are other fields that need to be added to the script , Otherwise, delete the specified Group The same job and instance Of Group Delete all . The above script assumes that all Group Labels All the same , So do not use it for Group Labels Inconsistent occasions .
5. Group Labels There is a prescribed order , Incorrect sequence will result in DELETE return 404. You have to try the right order yourself .
6. Use the following script to get all the current Group Labels, But the order of return does not satisfy the condition of order ：

baseurl=localhost:9091
curl -sS $baseurl/api/v1/metrics | jq -r '.data[].push_time_seconds.metrics[0].labels'