Summary: Prometheus storage

One 、 Introduce

prometheus Provides local storage （TSDB） Storage mode of sequential database , stay 2.0 After the version , The ability to compress data has been greatly improved （ Each sampling data only takes 3.5byte Left and right space ）, Single node can meet the needs of most users , But local storage hinders prometheus Implementation of clustering , Therefore, we should adopt Other temporal data to replace , such as influxdb.

prometheus It's divided into three parts , Namely ： Fetching the data 、 Store the data and Query data .

In the early days, there was a separate project called  TSDB, however , stay 2.1.x Some version of , This project is no longer maintained separately , Directly merge this project into prometheus It's on the trunk of .

Two 、 Case study

our prometheus At present, each business is stored separately , Store in the cloud （StorageClass）

Here is how to create StorageClass

  Two key documents ：

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: prometheus-data-03
provisioner: prometheus-data-03/nfs
reclaimPolicy: Retain

I understand ： Deploy a client pod, This client will be created according to the above StorageClass Push data to cloud storage

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-provisioner-03
spec:
  selector:
    matchLabels:
      app: nfs-provisioner-03
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-provisioner-03
    spec:
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-provisioner
          image: docker-registry.xxx.virtual/hubble/nfs-client-provisioner:v3.1.0-k8s1.11
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: prometheus-data-03/nfs
            - name: NFS_SERVER
              value: hubble-587ceb02-5e85e802.cnhz1.qfs.xxx.storage
            - name: NFS_PATH
              value: /hubble-wuhan-lkg
      volumes:
        - name: nfs-client-root
          nfs:
            server: hubble-587ceb02-5e85e802.cnhz1.qfs.xxx.storage
            path: /hubble-wuhan-lkg

above server The address is actually the address provided by cloud storage ：

Deploy prometheus：

kind: Service
apiVersion: v1
metadata:
  name: prometheus-headless
  namespace: example-nfs
  labels:
    app.kubernetes.io/name: prometheus
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app.kubernetes.io/name: prometheus
  ports:
  - name: web
    protocol: TCP
    port: 9090
    targetPort: web
  - name: grpc
    port: 10901
    targetPort: grpc
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: example-nfs

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus-example-nfs
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: example-nfs
roleRef:
  kind: ClusterRole
  name: prometheus
  apiGroup: rbac.authorization.k8s.io
---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
  namespace: example-nfs
  labels:
    app.kubernetes.io/name: prometheus
spec:
  serviceName: prometheus-headless
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        fsGroup: 1000
        runAsUser: 0
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - prometheus
            topologyKey: kubernetes.io/hostname
      containers:
      - name: prometheus
        image: docker-registry.xxx.virtual/hubble/prometheus:v2.34.0
        args:
        - --config.file=/etc/prometheus/config_out/prometheus.yaml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention.time=30d
        - --web.external-url=/example-nfs/prometheus
        - --web.enable-lifecycle
        - --storage.tsdb.no-lockfile
        - --storage.tsdb.min-block-duration=2h
        - --storage.tsdb.max-block-duration=1d
        ports:
        - containerPort: 9090
          name: web
          protocol: TCP
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /example-nfs/prometheus/-/healthy
            port: web
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        readinessProbe:
          failureThreshold: 120
          httpGet:
            path: /example-nfs/prometheus/-/ready
            port: web
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 30Gi
        volumeMounts:
        - mountPath: /etc/prometheus/config_out
          name: prometheus-config-out
          readOnly: true
        - mountPath: /prometheus
          name: prometheus-storage
        - mountPath: /etc/prometheus/rules
          name: prometheus-rules
      - name: thanos
        image: docker-registry.xxx.virtual/hubble/thanos:v0.25.1
        args:
        - sidecar
        - --tsdb.path=/prometheus
        - --prometheus.url=http://127.0.0.1:9090/example-nfs/prometheus
        - --reloader.config-file=/etc/prometheus/config/prometheus.yaml.tmpl
        - --reloader.config-envsubst-file=/etc/prometheus/config_out/prometheus.yaml
        - --reloader.rule-dir=/etc/prometheus/rules/
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - name: http-sidecar
          containerPort: 10902
        - name: grpc
          containerPort: 10901
        livenessProbe:
            httpGet:
              port: 10902
              path: /-/healthy
        readinessProbe:
          httpGet:
            port: 10902
            path: /-/ready
        volumeMounts:
        - name: prometheus-config-tmpl
          mountPath: /etc/prometheus/config
        - name: prometheus-config-out
          mountPath: /etc/prometheus/config_out
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
        - name: prometheus-storage
          mountPath: /prometheus
      volumes:
      - name: prometheus-config-tmpl
        configMap:
          defaultMode: 420
          name: prometheus-config-tmpl
      - name: prometheus-config-out
        emptyDir: {}
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
  volumeClaimTemplates:
  - metadata:
      name: prometheus-storage
      labels:
        app.kubernetes.io/name: prometheus
    spec:
      storageClassName: prometheus-data-03
      accessModes:
      - ReadWriteOnce
      volumeMode: Filesystem
      resources:
        requests:
          storage: 100Gi
        limits:
          storage: 300Gi

3、 ... and 、 Storage principle

prometheus according to block Block to store data , Every time 2 The hour is a unit of time , First, it will be stored in memory , When they arrive in 2 Hours later, , Will be automatically written to disk .

To prevent data loss caused by program exceptions , Adopted WAL Mechanism , namely 2 The data recorded within hours is stored in memory at the same time , A log will also be recorded , Stored in block Under the wal Directory . When the program starts again , Will wal The data in the directory is written to the corresponding block in , So as to achieve the effect of restoring data .

When deleting data , Delete entries will be recorded in tombstones in , Instead of deleting it immediately .

prometheus The storage method used is called “ Time is divided ”, Every block Is an independent database . The advantage is that it can improve query efficiency , Check the data of which time period , Just open the corresponding block that will do , No need to open redundant data .

Four 、 The data backup

1、 Full backup

　　 Backup prometheus Of data The directory can achieve the purpose of full backup , But less efficient .

2、 The snapshot backup

prometheus Provides a function , It's through API The way , Fast backup data . Realization way ：

• First , modify prometheus Start parameter of , Add the following two parameters ：

--storage.tsdb.path=/usr/local/share/prometheus/data \
--web.enable-admin-api

• then , restart prometheus
• Last , Call interface backup ：

#  Do not skip data in memory , That is, back up the data in memory at the same time 
curl -XPOST http://127.0.0.1:9090/api/v2/admin/tsdb/snapshot?skip_head=false
#  Skip data in memory 
curl -XPOST http://127.0.0.1:9090/api/v2/admin/tsdb/snapshot?skip_head=true

 skip_head effect ： Whether to skip the data stored in memory and not written to disk , Still in block The data in the block , The default is false

5、 ... and 、 Data restore

utilize api Made by snapshot after , Restore will snapshot The file in overwrites data Under the table of contents , restart prometheus that will do ！

Add a scheduled backup task （ Every Sunday 3 Point backup ）

crontable -e                              # Pay attention to the time zone , After modifying the time zone , Need to restart  crontab    systemctl  restart cron
 
0 3 * * 7 sudo /usr/bin/curl -XPOST -I http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot >> /home/bill/prometheusbackup.log