Istio Troubleshooting: uneven grpc service load

This article excerpts from istio Learning notes

The phenomenon

grpc call , The same client Your request always hits the same server Of pod, Cause uneven load .

analysis

grpc Is based on http2 Long connection of , Multiple requests to reuse the same connection . If not istio, Use only ordinary k8s service, Will not perceive grpc Agreed , Just as tcp To forward , Load balancing at the connection level , No load balancing at the request level . But in istio in , The default will be for grpc Load balancing at the request level , If uneven load is found , It is usually not configured correctly .

Must let grpc Load balancing at the request level , The core is to let istio Correct identification is grpc agreement , Don't configure it to tcp, use tcp Then load balancing can only be performed at the connection level , Request levels can be loaded unevenly .

resolvent

1. If you want to be exposed ,gateway in protocal To configure GRPC no need TCP, Example :

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: grpc-gw
  namespace: demo
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: grpc-demo-server
      number: 9000
      protocol: GRPC #  Use here  GRPC  no need  TCP

1. If you define vs, Need to use http Match without tcp, because grpc stay istio Chinese matching is also used http Field , Example :

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: grpc-svc
  namespace: demo
spec:
  gateways:
  - demo/grpc-gw
  hosts:
  - '*'
  http: #  Use here  http  no need  tcp
  - match:
    - port: 9000
    route:
    - destination:
        host: grpc.demo.svc.cluster.local
        port:
          number: 9000
      weight: 100

1. Deploying services service Of port name Need to use "grpc-" Opening definition , Give Way istio Be able to correctly identify , Example :

apiVersion: v1
kind: Service
metadata:
  name: grpc
  namespace: demo
spec:
  ports:
  - name: grpc-9000 #  With  grpc-  start 
    port: 9000
    protocol: TCP
    targetPort: 9000
  selector:
    app: grpc
  type: ClusterIP

For more protocol specification methods, please refer to istio Best practices : Explicitly specify the protocol for the service