Practice of hierarchical management based on kubesphere

author ： Xu Wei , R & D Engineer of aerospace network information

K8s Is a leader in container orchestration and distributed application deployment , stay K8s Environment , We only need to care about the business logic of the application , It reduces the management burden of our server network and storage . For a user ,K8s Is a very complex container orchestration platform , The cost of learning is very high .KubeSphere Abstracts the underlying K8s, And a high degree of productization , Build a full stack multi tenant container cloud platform , It provides a robust 、 Security 、 Rich in functions 、 With the ultimate experience Web Console , It's solved K8s High barriers to use and numerous cloud native ecological tools , Enables us to focus on rapid iterations of the business , Its multi-dimensional data monitoring , For the positioning of the problem , It's a great help .

Why are you in KuberSphere Realize hierarchical management on

stay KubeSphere in , Resources can be shared among tenants , According to the different roles assigned , Various resources can be operated . Between tenant and resource 、 There is a high degree of freedom between resources , Permission granularity is also relatively large . In our system , Resources have permission levels , For example, low-level users can invite 、 Give permission and other operations to operate high-level resources , Or like in a low-level project Pod It can be dispatched to high-level nodes , The resource . Such problems as cross level operation of resources , We are KubeSphere Based on this, the hierarchical management is realized .

What is a hierarchy

classification , As the name suggests, it is to decompose the whole according to the established standards 、 classification . We abstract it into a pyramid model , There are many levels from the foundation to the top of the tower , We use public resources as the foundation of the pyramid , With the highest authority admin As tower top , Other resources are divided into different levels according to permission levels . Low level resources cannot access high-level resources , A high-level resource can obtain all resources below its level , Constructed such a diminishing equity 、 A hierarchical system of hierarchical isolation .

How to realize hierarchical management

We define a label that represents the level kubernetes.io/level. Take a multi node cluster as an example , First of all, we will give users 、 Enterprise space 、 Nodes and other resources are labeled as representing levels . When inviting users to join an enterprise space or project , The level of enterprise space or project required to be added shall not be higher than the user's level , Similarly, when the project is bound to the enterprise space , It is also required that the level of the project shall not be higher than that of the enterprise space , To manage resources ; We believe that the level of resources under the same project is the same , Loads created based on the project 、Pod、 The level of services and other resources shall be consistent with the project ; meanwhile Pod Add node affinity to , In order to make Pod Dispatch to a node not higher than its permission level .

For example, here , We created a permission level that is 3 Users of demo-user, He can join with permission level no higher than 3 In the enterprise space or project .

kind: UserapiVersion: iam.kubesphere.io/v1alpha2metadata:  name: demo-user  labels:    kubernetes.io/level: 3spec:  email: [email protected]

Creating a permission level is 2 Project demo-ns, Then the load created based on the project 、Pod、 The permission level of storage and other resources is also 2.

apiVersion: v1kind: Namespacemetadata:   name: demo-ns   labels:     kubernetes.io/level: 2

be based on demo-ns The project creates a nginx Of Pod, His authority level is also 2, At the same time, add node affinity , It is required that the authority level shall not be higher than 2 Node .

apiVersion: apps/v1kind: Podmetadata:  labels:    kubernetes.io/level: 2  name: nginxspec:  containers:  - name: nginx    image: nginx    imagePullPolicy: IfNotPresent    ports:    - containerPort: 80      protocol: TCP  affinity:    nodeAffinity:      requiredDuringSchedulingIgnoredDuringExecution:       nodeSelectorTerms:        - matchExpressions:          - key: kubernetes.io/level            operator: Lt            values:            - 2        - matchExpressions:          - key: kubernetes.io/level            operator: In            values:            - 2

How to upgrade or downgrade resources

In the hierarchical management system , Support unlimited division of levels , Just define an intermediate value , You can insert a new level between the two levels , No need to manipulate other resources ; When upgrading or upgrading resources , You only need to modify the corresponding resource label label , You can upgrade or downgrade resources . Of course , When upgrading or degrading resources , We need to test the resources , Ensure that when upgrading , The permission level of its upper level resources shall not be lower than the target level ; meanwhile , When demoted , The permission level of its lower level resources shall not be higher than the target level . When the upgrade and downgrade operation conditions are not met , The corresponding resources need to be adjusted accordingly .

Between different levels Pod Network isolation

In the hierarchy , We require high-grade Pod Access to low-level Pod, But low grade Pod Cannot access high-level Pod, So how do we ensure that different levels Pod Network communication .

The project does not open network isolation ,Pod The network is interconnected , Therefore, a blacklist network strategy will be added here .

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: deny-all  namespace: demo-ns  labels:    kubernetes.io/level: 2spec:  podSelector: {}  policyTypes:  - Ingress

podSelector:{} Apply to all... In the project Pod, Prevent all flow from flowing in .

Then the release label level is greater than the target level （ Here is 2） The flow into （ We are right. Ingress The flow is not limited ）.

apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: level-match-network-policy  namespace: demo-ns  labels:    kubernetes.io/level: 2spec:  podSelector:    matchExpressions:    - key: kubernetes.io/level      operator: Gt      values:      - 2  policyTypes:  - Ingress

summary

KubeSphere It solves the problem of user building 、 Deploy 、 Pain points in management and observability , Its resources can be shared among multiple tenants . But in scenarios where resources have permission levels , Low level resources can operate high-level resources , Cause the problem of ultra vires management of resources . To solve this problem , We are KubeSphere On the basis of the transformation , To adapt to the hierarchical management between tenants and resources and between resources , At the same time, in the network strategy of the project , Add blacklist and whitelist strategies , Enhanced network isolation between projects , Make resource management more secure .

This article by the blog one article many sends the platform OpenWrite Release ！