Istio Troubleshooting: using istio to reserve ports causes pod startup failure

This article excerpts from istio Learning notes

Problem phenomenon

All newly started Pod unable ready,sidecar Report errors :

warning	envoy config	gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_15090: error adding listener: '0.0.0.0_15090' has duplicate address '0.0.0.0:15090' as existing listener

meanwhile istiod Also reported wrong :

ADS:LDS: ACK ERROR sidecar~172.18.0.185~reviews-v1-7d46f9dd-w5k8q.istio-test~istio-test.svc.cluster.local-20847 Internal:Error adding/updating listener(s) 0.0.0.0_15090: error adding listener: '0.0.0.0_15090' has duplicate address '0.0.0.0:15090' as existing listener

guess

Reading the newspaper wrong should be sidecar Get... At startup LDS The rules ,istiod Find out 0.0.0.0:15090 This monitor is repeated , It is an abnormal phenomenon , Send out xDS The rules will fail , Lead to sidecar It's never been possible ready.

analysis config_dump

Just find a normal that hasn't been restarted yet Pod, to glance at envoy config_dump:

kubectl exec debug-68b799694-n9q66 -c istio-proxy -- curl localhost:15000/config_dump

analysis json Find out static There is listening in the configuration 0.0.0.0:15090:

Positioning reason

Guess it is dynamic Also in the configuration 0.0.0.0:15090 The conflict caused by monitoring , and dynamic The listening source in is usually Kubernetes Service discovery of (Service, ServiceEntry), Check if there is Service monitor 15090:

kubectl get service --all-namespaces -o yaml | grep 15090

Finally, it was found that Service Yes 15090 port , Change to another port to recover .

Dig deep

Search for , You can find 15090 The port is istio Used to expose envoy prometheus The port of the indicator , yes envoy One of the ports used :

Reference resources Ports used by Istio .

But not all envoy All ports used are added to static Listening in configuration , Only 15090 and 15021 These two ports are static There is listening in the configuration , Also verified. Service Use 15021 Ports have the same problem .

Service Use others envoy The port of does not cause sidecar No ready The problem of , But at least make sure that the business program can not listen to these ports , Because I will follow envoy Conflict ,istio The official website also explains this : To avoid port conflicts with sidecars, applications should not use any of the ports used by Envoy.

Use advice

According to the above analysis , The following suggestions are made :

1. Service/ServiceEntry Don't define 15090 and 15021 port , Otherwise it will lead to Pod Failed to start successfully .
2. Business processes cannot listen envoy All ports used : 15000, 15001, 15006, 15008, 15020, 15021, 15090 .