Istio IV fault injection and link tracking

Previously, I introduced how to configure VirtualService Wait for the completion of traffic management , This blog will introduce Istio Another capability of is fault injection and timeout configuration , These configurations are also VirtualService Finish in .

Below VirtualService Pass through fault Field injected fault , At the same time VirtualService Configure how the upstream service returns 5xx Error code , Then retry 3 Time .

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: canary
spec:
  hosts:
    - canary
  http:
    - match:
        - headers:
            user:
              exact: jesse
      route:
        - destination:
            host: canary
            subset: v2
      ### retry if upstream server send 5xx
      retries:
        attempts: 3
        perTryTimeout: 2s
    - route:
      - destination:
          host: canary
          subset: v1
      ### send 500 to client in 80%
      fault:
        abort:
          httpStatus: 500
          percentage:
            value: 80

In addition, you can configure the timeout , Also in VeritualService Middle configuration .

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: canary
spec:
  hosts:
    - canary
  http:
    - match:
        - headers:
            user:
              exact: jesse
      route:
        - destination:
            host: canary
            subset: v2
      ### if upstream server response delay is greater than 1s, send timeout error to client
      timeout: 1s
    - route:
      - destination:
          host: canary
          subset: v1

In addition to fault injection and timeout settings ,VertualService Traffic mirroring can also be configured in , be used for A/B test , The introduced image traffic is due to the return Response header Added in Shadow Field , Therefore, it will not be really sent to users . In this way, without affecting users , Complete the test of the new version .

about VirtualService The rules in the , One thing to note ： When configuring multiple rules for the same target , Will follow in VirtualService Sequence execution in , That is, the first rule in the rule has the highest priority . So in VirtualService If it is found that the configured rules are not effective , You can check it out VirtualService All rules configured in , See if it is covered by the previous rules .

It says VertualService Various routing rule configurations in , So let's see Istio How to communicate with Jaeger Combine to complete link tracking .Jaeger It is a distributed link tracking system , In the design of the architecture Zipkin Architectural style , Both have many similar characteristics , Except for different development languages . As a rising star , be based on Go Powerful features , bring Jaeger In the field of cloud based primary ecology, it can be like a fish in water , Have a strong appeal , Even in the field of some new technology frameworks , As the default preferred Distributed Link Tracking System , Landing in various business scenarios . It supports cross platform 、 Diverse component tracking , for example ： Distributed edge routing component Traefik、Istio etc. . The following figure shows Jaeger The main components involved .

Client library （Client libraries）
Jaeger The client is OpenTracing API Language specific implementation of , They can be used manually or with existing OpenTracing Open source framework integration （ for example Flask,Dropwizard,gRPC etc. ） Together for distributed tracking applications . The detection service creates when it receives a new request Span, And the context information （Trace id,Span id and Baggage） Attach to outgoing request . Only id and baggage Propagate with the request ; All other profiling data （ Such as operation name , Time ,tag and log） Will not spread , It is asynchronously transmitted to Jaeger Back end . To minimize overhead ,Jaeger The client adopts various sampling strategies . for example ： When sampling a trace , Transfer the captured analysis range data to Jaeger Back end . When tracking sampling is not performed , No performance analysis data will be collected , And right OpenTracing API The call of will be short circuited , To minimize performance overhead . By default ,Jaeger The client to 0.1％ Of Traces sampling （ That is, every 1000 In the article 1 strip ）
agent （Agent）
Jaeger The agent is a network Daemons , It listens through UDP Sent Span, And then distribute it to the collector in batches , It is designed to be deployed to all hosts as a basic component . The agent abstracts the routing and discovery of the collector for the client .
The collector （Collector）
Jaeger Collector from Jaeger Agent receive tracking , And run them through processing pipes . The pipeline verifies the trace , Index it , Perform transformations and eventually store them .
Jaeger Storage （Storage）
Jaeger The storage of is a pluggable component , At present, we support Cassandra,Elasticsearch and Kafka.
Inquire about （Query）
A query is an item that retrieves a trace from storage and hosts it UI To display tracked services .
Ingester
Ingester Is a project from Kafka Topic Read and write to another storage backend （Cassandra,Elasticsearch） Service for .

It says Jaeger Basic components , Next, a practical example is given to demonstrate how to complete the link tracking of services . First, install on the cluster Jaeger, Here by pulling docker image The way to start Jaeger Of pod, Then it defines the method of listening to different ports Service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: istio-system
  labels:
    app: jaeger
spec:
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
      annotations:
        sidecar.istio.io/inject: "false"
        prometheus.io/scrape: "true"
        prometheus.io/port: "14269"
    spec:
      containers:
        - name: jaeger
          image: "docker.io/jaegertracing/all-in-one:1.23"
          env:
            - name: BADGER_EPHEMERAL
              value: "false"
            - name: SPAN_STORAGE_TYPE
              value: "badger"
            - name: BADGER_DIRECTORY_VALUE
              value: "/badger/data"
            - name: BADGER_DIRECTORY_KEY
              value: "/badger/key"
            - name: COLLECTOR_ZIPKIN_HOST_PORT
              value: ":9411"
            - name: MEMORY_MAX_TRACES
              value: "50000"
            - name: QUERY_BASE_PATH
              value: /jaeger
          livenessProbe:
            httpGet:
              path: /
              port: 14269
          readinessProbe:
            httpGet:
              path: /
              port: 14269
          volumeMounts:
            - name: data
              mountPath: /badger
          resources:
            requests:
              cpu: 10m
      volumes:
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: tracing
  namespace: istio-system
  labels:
    app: jaeger
spec:
  type: ClusterIP
  ports:
    - name: http-query
      port: 80
      protocol: TCP
      targetPort: 16686
    # Note: Change port name if you add '--query.grpc.tls.enabled=true'
    - name: grpc-query
      port: 16685
      protocol: TCP
      targetPort: 16685
  selector:
    app: jaeger
---
# Jaeger implements the Zipkin API. To support swapping out the tracing backend, we use a Service named Zipkin.
apiVersion: v1
kind: Service
metadata:
  labels:
    name: zipkin
  name: zipkin
  namespace: istio-system
spec:
  ports:
    - port: 9411
      targetPort: 9411
      name: http-query
  selector:
    app: jaeger
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-collector
  namespace: istio-system
  labels:
    app: jaeger
spec:
  type: ClusterIP
  ports:
    - name: jaeger-collector-http
      port: 14268
      targetPort: 14268
      protocol: TCP
    - name: jaeger-collector-grpc
      port: 14250
      targetPort: 14250
      protocol: TCP
    - port: 9411
      targetPort: 9411
      name: http-zipkin
  selector:
    app: jaeger

Then the code of the monitored service , here There is one tested demo Program ,demo There are three services service0,service1,service2. Request to service0 after , It will be delivered to service1,service1 Forward to service2.service2 Then return the request response . For link tracking , It is also necessary to bury the tested Application . How to bury it ？

Istio Agent can send automatically Span Information , So at the burying point, only some auxiliary means are needed to unify the whole tracking process . That is, the application propagates the information related to tracking by itself HTTP Header, So the agent is sending Span Information , We can correctly unify a trace . see demo Program , You can see service0 and service1 Will the request of the previous service header The message is sent to the next service . for example service1 The code in is as follows ： take request Medium header Information is added to the new NewRequest in , By visiting service2 Request , This way header The information in is passed to the next service .

  The application under test has passed Dockerfile Create a mirror image ,push To the image warehouse , So here are three deployment methods yaml File to complete the deployment , Here is service0 The file of ,service1 and service2 Of yaml The file only needs to be service0 Replace it .

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service0
  template:
    metadata:
      labels:
        app: service0
    spec:
      containers:
        - name: service0
          imagePullPolicy: Always
          image: cncamp/service0:v1.0
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: service0
spec:
  ports:
    - name: http-service0
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: service0

except service Of yaml file , also istio Of VirtualService and Gateway Configuration file information , From the configuration, you can see that the access entry is service0, The details are as follows

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: service0
spec:
  gateways:
    - service0
  hosts:
    - '*'
  http:
  - match:
      - uri:
          exact: /service0
    route:
      - destination:
          host: service0
          port:
            number: 80
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: service0
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http-service0
        number: 80
        protocol: HTTP

Finally, start the above object by command , And access the application , stay jaeger View link tracking information on .

### update tracing sampling
```sh
kubectl apply -f jaeger.yaml
kubectl edit configmap istio -n istio-system
set tracing.sampling=100
```
### Deploy tracing
```sh
kubectl create ns tracing
kubectl label ns tracing istio-injection=enabled
kubectl -n tracing apply -f service0.yaml
kubectl -n tracing apply -f service1.yaml
kubectl -n tracing apply -f service2.yaml
kubectl apply -f istio-specs.yaml -n tracing
```
### Check ingress ip
```sh
kubectl get svc -n istio-system
istio-ingressgateway   LoadBalancer   $INGRESS_IP
```
### Access the tracing via ingress for 100 times(sampling rate is 1%)
```sh
curl $INGRESS_IP/service0
```
### Check tracing dashboard
```sh
istioctl dashboard jaeger
```

stay Jaeger The information obtained on is as follows , It indicates that the link condition of the tested service is correctly collected and displayed .

The above example is only right Jaeger A simple introduction and use , more Jaeger Please check Official website Information .