Ebpf cilium practice (2) - underlying network observability

​ In the previous platform , There is no direct observability for the network flow between components , There is a problem in the communication between user components , Manual troubleshooting can only be performed through traditional command-line tools , and cilium Of Hubble Services can provide UI The interface shows users the real-time traffic status , At the same time, these indicators can be exposed to Prometheus Aggregate finishing , So that users can observe and monitor the underlying network state more intuitively .

Turn on Hubble UI service

cilium The observability of the network is determined by Hubble Services provide , In the installation cilium when , Not installed by default Hubble , You can turn on... With the following command Hubble service

helm upgrade cilium cilium/cilium --version 1.11.2 \   --namespace kube-system \   --reuse-values \   --set hubble.relay.enabled=true \   --set hubble.ui.enabled=true

After deployment , The status can be determined by the following command

$ kubectl get po -n kube-system |grep hubblehubble-relay-65ff5f9bf6-247pt         1/1     Running     0          5d19hhubble-ui-5f7cdc86c7-gq5hs            3/3     Running     0          5d19h$ kubectl get svc -n kube-system | grep hubblehubble-relay     ClusterIP   10.43.73.95    <none>        80/TCP                   5d19hhubble-ui        ClusterIP   10.43.20.190   <none>        80/TCP                   5d19h

Hubble After deployment , It is not directly accessible outside the cluster , You can open external access in the following ways

• Temporarily open

  When executing commands, you can use IP:12000 visit UI Interface , After exiting the command, you cannot continue to access

  cilium hubble ui

• For a long time

  adopt Rainbond How to add third-party components to the platform , Open or close at any time UI Access to the interface

  

  

  

Hubble UI Display information

Visit the main page

Enter the namespace you want to view , The current traffic topology is displayed in the middle of the page , The lower part shows the flow record

Click the traffic record to view the details

After clicking the component, only the relevant traffic is displayed

Select the information column displayed by the traffic record

Select the type of traffic to show

Select whether to ignore special types of traffic during display

docking Prometheus and Grafana

cilium Provides deployment Prometheus and Grafana Of yaml file , Which includes Grafana Template file , but cilium There are no open monitoring indicators when installed by default , Therefore, the monitoring indicators need to be enabled before deployment Prometheus and Grafana

Start monitoring indicators

helm upgrade cilium cilium/cilium --version 1.11.2 \   --namespace kube-system \   --reuse-values \   --set prometheus.enabled=true \   --set operator.prometheus.enabled=true \   --set hubble.enabled=true \   --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

Deploy Prometheus and Grafana

$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.11.2/examples/kubernetes/addons/prometheus/monitoring-example.yamlnamespace/cilium-monitoring createdserviceaccount/prometheus-k8s createdconfigmap/grafana-config createdconfigmap/grafana-cilium-dashboard createdconfigmap/grafana-cilium-operator-dashboard createdconfigmap/grafana-hubble-dashboard createdconfigmap/prometheus createdclusterrole.rbac.authorization.k8s.io/prometheus unchangedclusterrolebinding.rbac.authorization.k8s.io/prometheus unchangedservice/grafana createdservice/prometheus createddeployment.apps/grafana createddeployment.apps/prometheus created

Confirm the operation status

$ kubectl get po -n cilium-monitoringNAME                          READY   STATUS    RESTARTS   AGEgrafana-d69c97b9b-5ztrj       1/1     Running   0          5d20hprometheus-655fb888d7-456n4   1/1     Running   0          5d20h$ kubectl get svc -n cilium-monitoringNAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGEgrafana      ClusterIP   10.43.230.15    <none>        3000/TCP   5d20hprometheus   ClusterIP   10.43.219.180   <none>        9090/TCP   5d20h

Open external access

• Temporarily open

  kubectl -n cilium-monitoring port-forward service/grafana --address 0.0.0.0 --address :: 3000:3000kubectl -n cilium-monitoring port-forward service/prometheus --address 0.0.0.0 --address :: 9090:9090

• For a long time

  

  

  

  

Grafana Display information

Cilium Metrics

Cilium Operator

Hubble