The Coolest Kubernetes Network Solution Cilium Getting Started Tutorial

The industry's most widely usedK8S CNI网络方案 Calico 宣布支持 eBPF,And as the first pass eBPF 实现了 kube-proxy 所有功能的 K8S 网络方案——Cilium,Does its prescient name turn into an advantage?,继而成为 CNI What about the new headliner?？Let's get started today Cool Kubernetes 网络方案 Cilium.

Cilium介绍

以下基于 CiliumOfficial website document translation.

当前趋势

The application system of modern data center has gradually turned to the development system based on microservice architecture,A microservice architecture application system is composed of multiple small independent services,between them through lightweight communication protocols such as HTTP、gRPC、Kafka 等进行通信.Services under the microservice architecture naturally have the characteristics of dynamic changes,结合容器化部署,Often causes large-scale container instances to start or restart.To ensure safe reachability between such highly dynamic microservice applications,既是挑战,也是机遇.

现有问题

传统的 Linux 网络访问安全控制机制（如 iptables）是基于静态环境的IP地址和端口配置网络转发、过滤等规则,但是 IP 地址在微服务架构下是不断变化的,非固定的;出于安全目的,协议端口(例如 HTTP 传输的 TCP 端口 80)也不再固定用来区分应用系统.为了匹配大规模容器实例快速变化的生命周期,传统网络技术需要维护成千上万的负载均衡规则和访问控制规则,并且需要以不断增长的频率更新这些规则,而如果没有准确的可视化功能,要维护这些规则也是十分困难,这些对传统网络技术的可用性和性能都是极大的挑战.比如经常会有人对 kube-proxy 基于 iptables 的服务负载均衡功能在大规模容器场景下具有严重的性能瓶颈,同时由于容器的创建和销毁非常频繁,基于 IP 做身份关联的故障排除和安全审计等也很难实现.

解决方案

Cilium 作为一款 Kubernetes CNI 插件,从一开始就是为大规模和高度动态的容器环境而设计,并且带来了 API 级别感知的网络安全管理功能,通过使用基于 Linux 内核特性的新技术——BPF,提供了基于 service/pod/container 作为标识,而非传统的 IP 地址,来定义和加强容器和 Pod 之间网络层、应用层的安全策略.因此,Cilium 不仅将安全控制与寻址解耦来简化在高度动态环境中应用安全性策略,而且提供传统网络第 3 层、4 层隔离功能,以及基于 http 层上隔离控制,来提供更强的安全性隔离.

另外,由于 BPF 可以动态地插入控制 Linux 系统的程序,实现了强大的安全可视化功能,而且这些变化是不需要更新应用代码或重启应用服务本身就可以生效,因为 BPF 是运行在系统内核中的.

以上这些特性,使 Cilium 能够在大规模容器环境中也具有高度可伸缩性、可视化以及安全性.
 

部署 Cilium

部署 Cilium 非常简单,可以通过单独的 yaml File deployment of all components（Currently I have deployed this way1.7.1 版本）,也可以通过 helm chart 一键完成.What matters is the deployment environment and timing：

1. It is officially recommended that all deployment nodes use Linux Latest stable kernel version,so that all functions are enabled,Specific deployment environment recommendations can be found here.

2. 作为一个 Kubernetes 网络组件,it should be deployed Kubernetes After other basic components,to deploy.这里,The problem I have myself is,因为还没有 CNI 插件,coredns The state of the component is always pending的,until deployed Cilium 后,coredns Completed reset becomesrunning状态.

下图是 Cilium The overall deployment component diagram of：

测试安装效果

官方提供了一个 connectivity 检查工具,to detect the deployed Cilium 是否工作正常.If you have some restrictions on the network environment,I made some simple modifications,可以参照这里.部署起来很简单,Make sure you have at least two nodes available,Otherwise there are several deployment will not work successfully：

> kubectl apply -f connectivity-check.yaml

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
echo-a                                  1/1     1            1           16d
echo-b                                  1/1     1            1           16d
host-to-b-multi-node-clusterip          1/1     1            1           16d
host-to-b-multi-node-headless           1/1     1            1           16d
pod-to-a                                1/1     1            1           16d
pod-to-a-allowed-cnp                    1/1     1            1           16d
pod-to-a-external-1111                  1/1     1            1           16d
pod-to-a-l3-denied-cnp                  1/1     1            1           16d
pod-to-b-intra-node                     1/1     1            1           16d
pod-to-b-multi-node-clusterip           1/1     1            1           16d
pod-to-b-multi-node-headless            1/1     1            1           16d
pod-to-external-fqdn-allow-google-cnp   1/1     1            1           16d

如果所有的 deployment can run successfully,说明 Cilium has been deployed successfully and works fine.

Network visualization artifact Hubble

上文提到了 Cilium The power is to provide a simple and efficient network visualization function,它是通过 Hubble组件完成的.Cilium在1.7Released and open sourcedHubble,It is specially designed for network visualization,能够利用 Cilium 提供的 eBPF 数据路径,获得对 Kubernetes 应用和服务的网络流量的深度可见性.这些网络流量信息可以对接 Hubble CLI、UI 工具,可以通过交互式的方式快速诊断如与 DNS 相关的问题.除了 Hubble 自身的监控工具,还可以对接主流的云原生监控体系—— Prometheus 和  Grafana,实现可扩展的监控策略.

部署 Hubble 和 Hubble UI

官方提供了基于 Helm Chart 部署方式,This allows flexible control over deployment variables,Implement different monitoring strategies.out of wanting to try hubble UI 和对接 Grafana,I deployed like this：

> helm template hubble \
    --namespace kube-system \
    --set metrics.enabled="{dns:query;ignoreAAAA;destinationContext=pod-short,drop:sourceContext=pod;destinationContext=pod,tcp,flow,port-distribution,icmp,http}" \
    --set ui.enabled=true \
    > hubble.yaml
> kubectl apply -f hubble.yaml
# 包含两个组件
# - daemonset hubble
# - deployment hubble UI
> kubectl get pod -n kube-system |grep hubble
hubble-67ldp                       1/1     Running   0          21h
hubble-f287p                       1/1     Running   0          21h
hubble-fxzms                       1/1     Running   0          21h
hubble-tlq64                       1/1     Running   1          21h
hubble-ui-5f9fc85849-hkzkr         1/1     Running   0          15h
hubble-vpxcb                       1/1     Running   0          21h 

运行效果

由于默认的 Hubble UI 只提供了 ClusterIP 类似的 service,Not accessible from outside.因此需要创建一个 NodePort 类型的 service,如下所示：

# hubble-ui-nodeport-svc.yaml
kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: hubble-ui-np
spec:
  selector:
    k8s-app: hubble-ui
  ports:
    - name: http
      port: 12000
      nodePort: 32321
  type: NodePort

执行 kubectl apply -f hubble-ui-nodeport-svc.yaml,through any cluster node IP 地址加上 32321 端口访问 Hubble UI 的 web 服务了.The opening effect is as follows：

• Page on the upper part is a set of before deployment conectivity-check Component data flow diagram,官方叫做 Service Map,Network-based auto-discovery by default 3 层和 4 Layer access dependency path,看上去非常 cool,It also feels a bit distributed link tracing graph.click on a service,You can also see a more detailed relationship diagram：

• 下图是 kube-system Data flow diagram under namespace,能看到 Hubble-UI 组件和 Hubble 组件是通过gRPC 进行通信的,非常有趣.But it is curious that,为何没有显示 Kubernetes Core components between the invocation of the diagram:

The lower part of the page is the default display data for each of the detailed description of the flow path,including the requesting pod 名称、发起请求的 service 名称、请求目标的 pod 名称、请求目标的 service 名称、目标 IP、目标端口、目标 7 层信息、请求状态、Last viewed time etc.,如下图所示：

Click on any one flow,To view more detailed information：

The lower part of the page can be switched to display by clicking network policy 模式,Lists all network policies in the current namespace：

如果想开启网络 7 层的可视化观察,就需要对目标 pod 进行 annotations ,感兴趣可以看这里,Not detailed in the introductory chapter.

Is this kind of network visualization your dream?,It can definitely help a lot when troubleshooting request invocation problems.

对接 Grafana + Prometheus

if you are the same Grafana+ Prometheus 的忠实粉丝,那么使 Hubble Docking them is the necessary operation.Careful classmates have discovered before helm template the mystery：

--set metrics.enabled="{dns:query;ignoreAAAA;destinationContext=pod-short,drop:sourceContext=pod;destinationContext=pod,tcp,flow,port-distribution,icmp,http}"
# 上面的设置,表示开启了 hubble 的 metrics 输出模式,and output the above information.
# 默认情况下,Hubble daemonset Automatically exposed metrics API 给 Prometheus.

You can dock existing Grafana+Prometheus 服务,It is also possible to deploy a simple：

# The following command will run in the namespace cilium-monitoring 下部署一个 Grafana 服务和 Prometheus 服务
$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.6/examples/kubernetes/addons/prometheus/monitoring-example.yaml
# 创建对应 NodePort Service,方便外部访问 web 服务
$ kubectl expose deployment/grafana --type=NodePort --port=3000 --name=gnp -n cilium-monitoring 
$ kubectl expose deployment/prometheus --type=NodePort --port=9090 --name=pnp -n cilium-monitoring

完成部署后,打开 Grafana 网页,Import official production dashboard,可以快速创建基于 Hubble 的 metrics 监控.等待一段时间,就能在 Grafana I saw the data：

Cilium 配合 Hubble,的确非常好用！