Thanos store component

How to read the data uploaded to the object storage ? It provides store gateway The components of ,store gateway Simply read the data stored in the object , through store api Exposed to the query,query Just call store api Then get the data , In this way, the data in the object storage can be obtained , In this way, you can query the data of a long time ago .

query It can be implemented to multiple back ends storeapi Component query data , This storeapi Is to expose data to query To query .

Some Prometheus The instance is up , Other instances are not hung , In this way, you can still query the data .

 

Store Components

---

We have installed Thanos Of Sidecar and Querier Components , It can already be done Prometheus High availability , adopt Querier Provide a unified entry to query multiple Prometheus Monitoring data , And it can automatically de duplicate the monitoring data , But it's also very important that the object store has not been configured , If you want to view historical monitoring data, you can't , At this time, we need to configure Thanos Store Components , Store historical monitoring indicators in the object store .（2 The data after hours is not uploaded to the object store , If you want to view historical monitoring data, you need to configure store Components , The previous configuration is 6 Hours ）

at present Thanos Supported object stores are ：

To be used in a production environment, it is best to use Stable State of , such as S3 Or compatible S3 Service for , such as Ceph、Minio wait .

For domestic users, of course, the most convenient way is to use alicloud directly OSS Or Tencent cloud COS Such a service , But most of the time, our services may not run on the public cloud , So here we use Minio To deploy a compatible S3 Protocol object storage service .

install Minio

---

MinIO  It's based on Apache License v2.0 High performance distributed object storage service based on open source protocol , Designed for large-scale private cloud infrastructure . It's Amazon compatible S3 Cloud storage service interface , It is very suitable for storing large capacity unstructured data , For example, pictures 、 video 、 Log files 、 Backup data and containers / Virtual machine image, etc , And an object file can be any size , From several kb To maximum 5T Unequal .

To install Minio It's very easy , Again, we're going to Minio The installation to Kubernetes In the cluster , You can refer to official documents directly   Use Kubernetes Deploy MinIO, stay Kubernetes The cluster can be deployed independently 、 Distributed or shared modes , It can be deployed according to the actual situation , Here we just test with the simplest independent mode .

Directly use the following Deployment To manage Minio Service for ：

# minio-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: minio
    spec:
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-pvc
      containers:
        - name: minio
          volumeMounts:
            - name: data
              mountPath: "/data"
          image: minio/minio:latest
          args: ["server", "--console-address", ":9001", "/data"]
          env:
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
            - containerPort: 9001
          readinessProbe:
            httpGet:
              path: /minio/health/ready
              port: 9000
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /minio/health/live
              port: 9000
            initialDelaySeconds: 10
            periodSeconds: 10

Because the new version of the image distinguishes Console and API Ports of two services , So we need to go through  --console-address  Parameter to specify Console Port of service , default API The service is 9000 On port .

And then through a name called  minio-pvc  Of PVC Objects persist data ：

# minio-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10G
  storageClassName: longhorn #  Best use LocalPV

  In the end, we can go through NodePort Type of Service The service will Minio Exposed to external users ：

# minio-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  ports:
    - name: console
      port: 9001
      targetPort: 9001
      nodePort: 30091
    - name: api
      port: 9000
      targetPort: 9000
  selector:
    app: minio
  type: NodePort

Create the above resource object directly ：

 * kubectl apply -f https://p8s.io/docs/thanos/manifests/minio.yaml

After successful deployment , It can be accessed in the browser  http://<nodeip>:30091  Access to the MinIO Console service , Through the  MINIO_ACCESS_KEY  and  MINIO_SECRET_KEY  To log in ：

Then create a new one called  thanos  Of bucket bucket ：

install Thanos Store

---

Now the object store is ready , We can deploy Store Component , This component is stored from the object to Querier Provide metrics data .

According to the... Created above Minio Create an object storage configuration file as shown below ：

# thanos-storage-minio.yaml
type: s3
config:
  bucket: thanos
  endpoint: minio.default.svc.cluster.local:9000
  access_key: minio
  secret_key: minio123
  insecure: true
  signature_version2: false

  Use the configuration file above to create a Secret object ：

 * kubectl create secret generic thanos-objectstorage --from-file=thanos.yaml=thanos-storage-minio.yaml -n kube-mon

The above is the configuration file of the docking object storage  

  Then create Store Component's resource manifest file , One thing to note here is to add a  thanos-store-api: "true"  The label of , So what we created earlier  thanos-store-gateway  This Headless Service You can automatically discover this service ,Querier When a component queries data, it can use Sidecar To get data, you can also use this Store Components go to the object store to get data .（ This label tells querystore Components also implement store-api,query Component docking has thanos-store-api The components of , You can go through query Component query store Data inside , You can also check sidecar data ）

Below it are all the matching implementations thanos-store-api Of pod Connect . 

  The above Secret Object passing Volume Form attached to the container  /etc/secret  Under the table of contents , adopt  objstore.config-file  Parameters can be specified ：

# thanos-store.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store-gateway
  namespace: kube-mon
  labels:
    app: thanos-store-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app: thanos-store-gateway
  serviceName: thanos-store-gateway
  template:
    metadata:
      labels:
        app: thanos-store-gateway
        thanos-store-api: "true"  // Tell the system the current store Components also implement store-api, that query You can query store Data inside , You can also check sidecar The data in it 
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - thanos-store-gateway
      containers:
        - name: thanos
          image: thanosio/thanos:v0.25.1
          args:
            - "store"
            - "--log.level=debug"
            - "--data-dir=/data"
            - "--objstore.config-file=/etc/secret/thanos.yaml"
            - "--index-cache-size=500MB"
            - "--chunk-pool-size=500MB"
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
          livenessProbe:
            httpGet:
              port: 10902
              path: /-/healthy
          readinessProbe:
            httpGet:
              port: 10902
              path: /-/ready
          volumeMounts:
            - name: object-storage-config
              mountPath: /etc/secret
              readOnly: false
            - mountPath: /data
              name: data
      volumes:
        - name: object-storage-config
          secret:
            secretName: thanos-objectstorage
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: longhorn
        resources:
          requests:
            storage: 1Gi

Store Gateway In fact, a certain degree of Statelessness can be achieved , It will need a little disk space to index the object store to speed up the query , But data is not that important , It can be deleted , After deletion, it will automatically pull the object storage query data and re index , To avoid re indexing every time you restart , So use StatefulSet Deploy Store Gateway, Mount a small capacity PV. Deploy two copies , Can achieve Store Gateway High availability .

Then directly create the above resource object ：（ You can see thanos-store-api=true Of the label pod Yes 4 individual , that query When the component is docked, it should be docked to this 4 individual pod It's gone. ）

 * kubectl apply -f https://p8s.io/docs/thanos/manifests/thanos-store.yaml
 * kubectl get pods -n kube-mon -l thanos-store-api=true
NAME                     READY   STATUS    RESTARTS      AGE
prometheus-0             2/2     Running   0             20m
prometheus-1             2/2     Running   0             21m
thanos-store-gateway-0   1/1     Running   0             24m
thanos-store-gateway-1   1/1     Running   0             24m

  When the deployment is successful, go to Thano Of Querier View... On the page Store Information can be found here we configure the Store Component ：

Come here to prove our Store The component is also configured successfully . But another obvious problem is that we only configure to query data in the object store , Where to write data to object storage ？ Of course, it's still Sidecar It's in the component . So again, we need to put  objstore.config-file  Parameters and Secret Objects should also be configured to Sidecar In components ：（ The data of the object has not been uploaded to the object store , Just to create this store Components , Who sent the data to store What about components? ？sidecar Upload data to the object store , So it needs to be in sidecar The configuration is uploaded to the object store ）

......
volumes:
- name: object-storage-config
  secret:
    secretName: thanos-objectstorage
......
args:
- sidecar
- --log.level=debug
- --tsdb.path=/prometheus
- --prometheus.url=http://localhost:9090
- --reloader.config-file=/etc/prometheus/prometheus.yaml.tmpl
- --reloader.config-envsubst-file=/etc/prometheus-shared/prometheus.yaml
- --reloader.rule-dir=/etc/prometheus/rules/
- --objstore.config-file=/etc/secret/thanos.yaml
......
volumeMounts:
- name: object-storage-config
  mountPath: /etc/secret
  readOnly: false
......

When sidecar Read the object store we configured if it is found locally Prometheus The data directory contains tsdb The data block is uploaded to the object storage . Every time 2 Hours to generate new data blocks , If there is a new data block, it will be uploaded .

Update again after configuration Sidecar Components , After the configuration takes effect, data will be transferred to MinIO It went to the （ There are more than two hours of TSDB Piece of data ）, We can go MinIO View validation on the page of ： 

Then the future historical data will not be lost .