Deeply understand the fusing configuration of istio traffic management

Environmental preparation

httpbin service

httpbin It's a use Python + Flask Compiling HTTP Request response service .

The service is mainly used for testing HTTP library , You can send him a request , Then he will return your request according to the specified rules .

Deploy httpbin service ：

kubectl apply -f samples/httpbin/httpbin.yaml

fortio service

Fortio It's a microservice （http, grpc) Load test library , Command line tools , Advanced echo server and web UI stay go (golang).Fortio Allows you to specify a set of query loads per second , And record the delay histogram and other useful statistical data .

Deploy fortio service ：

kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml

httpbin The service is the server that receives the request , fortio The service is the client that sends the request .

Fuse configuration

Create target rules , visit httpbin Apply fusing configuration during service ：

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
EOF

stay fortio Service oriented httpbin The service sends concurrent requests ：

export FORTIO_POD=$(kubectl get pod -l app=fortio -n istio-demo -o jsonpath={
     .items..metadata.name})
kubectl exec -it "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 5 -loglevel Warning http://httpbin:8000/get

The results are as follows ：

I0725 11:43:55.895950 4039039 request.go:621] Throttling request took 1.130727121s, request: GET:https://172.24.28.71:6443/apis/ceph.rook.io/v1?timeout=32s
03:43:58 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.17.1 running at 0 queries per second, 104->104 procs, for 5 calls: http://httpbin:8000/get
Starting at max qps with 5 thread(s) [gomax 104] for exactly 5 calls (1 per thread + 0)
03:43:58 W http_client.go:806> [3] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [1] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [2] Non ok http code 503 (HTTP/1.1 503)
Ended after 5.398808ms : 5 calls. qps=926.13
Aggregated Function Time : count 5 avg 0.0035955544 +/- 0.001035 min 0.002619591 max 0.005337051 sum 0.017977772
# range, mid point, percentile, count
>= 0.00261959 <= 0.003 , 0.0028098 , 60.00, 3
> 0.004 <= 0.005 , 0.0045 , 80.00, 1
> 0.005 <= 0.00533705 , 0.00516853 , 100.00, 1
# target 50% 0.0029049
# target 75% 0.00475
# target 90% 0.00516853
# target 99% 0.0053202
# target 99.9% 0.00533537
Sockets used: 5 (for perfect keepalive, would be 5)
Jitter: false
Code 200 : 2 (40.0 %)
Code 503 : 3 (60.0 %)
Response Header Sizes : count 5 avg 92 +/- 112.7 min 0 max 230 sum 460
Response Body/Total Sizes : count 5 avg 419.8 +/- 177.3 min 275 max 637 sum 2099
All done 5 calls (plus 0 warmup) 3.596 ms avg, 926.1 qps

You can see , Sending 5 A request , Yes 3 A request returns 503 .

see Envoy journal

Execute the following command , see fortio Service Envoy journal ：

kubectl logs -l app=fortio -c istio-proxy

You can see fortio The service is right httpbin Log of service invocation ：

{
    
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.146.239:8000",
     "downstream_remote_address": "172.24.158.166:55030",
     "duration": 1,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "via_upstream",
     "response_flags": "-",
     "route_name": "allow_any",
     "start_time": "2022-07-22T09:19:04.465Z",
     "upstream_cluster": "PassthroughCluster",
     "upstream_host": "172.24.146.239:8000",
     "upstream_local_address": "172.24.158.166:55042",
     "upstream_service_time": "1",
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}

among ,response_flags by -, Nothing unusual .

Execute the following command , see httpbin Service Envoy journal ：

kubectl logs -l app=httpbin -c istio-proxy

You can see httpbin Service by fortio Called by the service Envoy journal ：

{
    
     "authority": "httpbin:8000",
     "bytes_received": 0,
     "bytes_sent": 81,
     "connection_termination_details": null,
     "downstream_local_address": "172.24.158.96:80",
     "downstream_remote_address": "172.24.158.166:55042",
     "duration": 0,
     "method": "GET",
     "path": "/get",
     "protocol": "HTTP/1.1",
     "request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
     "requested_server_name": null,
     "response_code": 503,
     "response_code_details": "upstream_reset_before_response_started{overflow}",
     "response_flags": "UO",
     "route_name": "default",
     "start_time": "2022-07-22T09:19:04.466Z",
     "upstream_cluster": "inbound|80||",
     "upstream_host": null,
     "upstream_local_address": null,
     "upstream_service_time": null,
     "upstream_transport_failure_reason": null,
     "user_agent": "fortio.org/fortio-1.17.1",
     "x_forwarded_for": null
}

among ,response_flags by UO, Indicates the upstream caused by fusing （upstream） overflow .

In depth analysis

adopt Envoy journal , We can make some analysis and judgment ：

When httpbin Service Request OK When , The calling process is as follows ：

When httpbin Service The fuse is triggered When , The calling process is as follows ：

When httpbin When the service fuse is triggered , return 503 The status code is httpbin Service Envoy.

clear

kubectl delete destinationrule httpbin
kubectl delete -f samples/httpbin/httpbin.yaml
kubectl delete -f samples/httpbin/sample-client/fortio-deploy.yaml

Last , Thank you for being so handsome , Give it back to me. give the thumbs-up .