Vivo large scale kubernetes cluster automation operation and maintenance practice

author ：vivo  Internet server team -Zhang Rong

One 、 background

With vivo Business migration to K8s The growth of , We need to K8s Deploy to multiple data centers . How to be efficient 、 Reliably manage multiple large-scale data centers K8s Clustering is the key challenge we face .kubernetes The node of needs to be aligned to OS、Docker、etcd、K8s、CNI And network plug-in installation and configuration , Maintaining these dependencies is tedious and error prone .

In the past, the deployment and capacity expansion of clusters were mainly through ansible Choreograph the task , Black screen operation 、 Configure cluster inventory and vars perform ansible playbook. The main difficulties of cluster operation and maintenance are as follows ：

• Manual black screen cluster O & M operation is required , There are operational errors and cluster configuration differences .

• The deployment scripting tool has no specific version control , It is not conducive to cluster upgrade and configuration change .

• Deploying scripts online takes a lot of time to validate , There are no specific test cases and CI verification .

• ansible The task is not split into modular installations , It should be broken into parts . Specific to the K8s、etcd、addons The role of modular management , Can be performed separately ansible Mission .

• Mainly through binary deployment , You need to maintain a cluster management system . The deployment process is cumbersome , Low efficiency .

• The parameter management of components is chaotic , Specify parameters from the command line .K8s The maximum number of components in the 100 The above parameter configuration . Every iteration of the big version changes .

This article will share our Kubernetes-Operator, use K8s The declarative form of API Design , Cluster administrators and Kubernetes-Operator Of CR Interact with resources , Simplify 、 Reduce task risk . Just one cluster administrator can maintain thousands of K8s node .

Two 、 Cluster deployment practice

2.1  Introduction to cluster deployment

Based mainly on ansible Defined OS、Docker、etcd、k8s and addons And other cluster deployment tasks .

The main process is as follows ：

• Bootstrap OS

• Preinstall step

• Install Docker

• Install etcd

• Install Kubernetes Master

• Install Kubernetes node

• Configure network plugin

• Install Addons

• Postinstall setup

The above is the key process of cluster one click deployment . When deployed in multiple data centers K8s After cluster , For example, the security vulnerability of cluster components 、 Launch of new functions 、 When the online cluster is changed, such as component upgrade , It needs to be handled carefully . We have broken the whole into parts , Deal with a single module . Avoid full implementation ansible Script , Increase the difficulty of maintenance . For example Docker、etcd、K8s、network-plugin and addons Modular management and operation and maintenance , Separate shall be provided ansible Script entry , More sophisticated operation and maintenance , Covering most of the lifecycle management of the cluster . meanwhile kubernetes-operator Of api The corresponding operation can be easily selected during design yml To perform operations .

The cluster deployment optimization operation is as follows ：

（1）K8s The component parameter management of is through ConmponentConfig[

1

] Provided API To identify the configuration file .

• 【 Maintainability 】 When the component parameter exceeds 50 Configuration becomes difficult to manage when there are more than .

• 【 Scalability 】 For upgrades , Versioned configuration parameters are easier to manage . Because the parameters of a large version of the community have not changed .

• 【 Programmability 】 Components can be （JSON/YAML） Object . If you enable dynamic kubelet configuration option , The modified parameters will take effect automatically , No need to restart the service .

• 【 Configurability 】 Many types of configurations cannot be represented as key-value form .

（2） Plan to switch to kubeadm Deploy

• Use kubeadm Yes K8s Life cycle management of clusters , Reduce the cost of maintaining the cluster .

• Use kubeadm Certificate management for , If the certificate is uploaded to secret Reduce the time consumption of the certificate in the host copy and regenerate the certificate .

• Use kubeadm Of kubeconfig Generate admin kubeconfig file .

• kubeadm Other functions such as image management 、 Configuration center upload-config、 Automatically label and stain control nodes .

• install coredns and kube-proxy addons.

（3）ansible Use standard

• Use ansible The built-in module handles the deployment logic .

• Avoid using hostvars.

• Avoid using delegate_to.

• Enable –limit  Pattern .

• wait .

2.2 CI  Matrix test

Deployed cluster , A lot of scenario tests and simulations are required . Ensure the reliability and stability of online environment changes .

CI The test cases of matrix part are as follows .

（1） Grammar test ：

• ansible-lint

• shellcheck

• yamllint

• syntax-check

• pep8

（2） Cluster deployment test ：

• Deployment cluster

• Expansion and contraction control node 、 Computing node 、etcd

• Upgrade cluster

• etcd、Docker、K8s and addons Parameter change, etc

（3） Performance and functional tests ：

• Check kube-apiserver Whether it works properly

• Check whether the network between nodes is normal

• Check whether the calculation node is normal

• K8s e2e test

• K8s conformance  test

• Other tests

So this is taking advantage of GitLab、gitlab-runner[

2

]、ansible and kubevirt[

3

] And other open source software CI technological process .

The detailed deployment steps are as follows ：

• stay K8s Cluster deployment gitlab-runner, And docking GitLab Warehouse .

• stay K8s Cluster deployment Containerized-Data-Importer (CDI)[
  4
  ] Components , Used to create pvc The image file that stores the virtual machine .

• stay K8s Cluster deployment kubevirt, For creating virtual machines .

• Write in the code warehouse gitlab-ci.yaml[
  5
  ],  Plan the cluster test matrix .

As shown in the figure above , When developers are in GitLab Submit PR Will trigger a series of operations . Here we mainly show the creation of virtual machines and cluster deployment . In fact, syntax checking and performance testing are also deployed in our cluster gitlab-runner, Through these gitlab-runner establish CI Of job To carry out CI technological process .

Specifically CI The process is as follows ：

• Developers submit PR.

• Trigger CI Automatically ansible Syntax check .

• perform ansible Script to create namespace,pvc and kubevirt Virtual machine template for , Finally, the virtual machine is K8s Up operation . It's mainly used here ansible Of K8s modular [
  6
  ] To manage the creation and destruction of these resources .

• call ansible Script to deploy K8s colony .

• After the cluster is deployed, perform functional verification and performance test .

• The destruction kubevirt、pvc And so on . Delete the virtual machine , Release resources .

As shown in the figure above , When developers submit multiple PR when , Will be in K8s Create multiple job, Every job Will execute the above CI test , It doesn't affect each other . This mainly uses kubevirt The ability of , Realized K8s on K8s The architecture of .

kubevirt The main abilities are as follows ：

• Provide standard K8s API, adopt ansible Of K8s Modules can manage the lifecycle of these resources .

• Reuse the K8s Scheduling capability of , Resources are managed and controlled .

• Reuse the K8s Network capabilities , With namespace Isolation , Each cluster network does not affect each other .

3、 ... and 、Kubernetes-Operator  practice

3.1 Operator  Introduce

Operator Is a controller for a specific application , You can extend the  K8s API The function of , To represent the K8s User created 、 Examples of configuring and managing complex applications . be based on K8s Resource and controller concept building , It also covers the knowledge of specific fields or applications . Used to automate the application lifecycle it manages .

summary  Operator Function as follows ：

• kubernetes controller

• Deploy or manage an application , Such as a database 、etcd etc.

• User defined application lifecycle management

• Deploy

• upgrade

• Expansion and contraction capacity

• Backup

• Self repair

• wait

3.2 Kubernetes-Operator CR  Introduce

kubernetes-operator The use of many custom CR Resources and controllers , Here is a brief introduction to functions and functions .

【ClusterDeployment】

:   The only one configured by the administrator CR, among MachineSet、Machine and Cluster Its sub resources or associated resources .ClusterDeployment It is the entry of all configuration parameters , Defined as etcd、K8s、lb、 Cluster version 、 Network and addons And so on .

【MachineSet】

： The set of clustered roles includes the control nodes 、 Compute nodes and etcd Configuration and execution status of .

【Machine】

： Specific information about each machine , Including the roles to which they belong 、 Node information and execution status .

【Cluster】

： and ClusterDeployment Corresponding , its status Defined as subresource, Reduce

clusterDeployment Triggering pressure of . Mainly used for storage ansible The status of the script executed by the executor .

【ansible actuator 】

： It mainly includes K8s Self job、configMap、Secret And self-developed job controller . among job It's mainly used to perform ansible Script for , because K8s Of job The state of success and failure , such job  The controller is well observed ansible The success or failure of execution , At the same time, you can also go through job Corresponding pod Log to view ansible Detailed execution process of .configmap Mainly used for storage ansible Execution time dependent inventory And variables , Hang on to job On .secret It mainly stores the key of the login host , It is also attached to job On .

【 Expansion controller 】

： Additional controller mainly used to extend the functions of cluster management , In the deployment kubernetes-operator We made it customized , You can select the expansion controller you need . such as addons The controller is mainly responsible for addon Plug in installation and management .clusterinstall The main generation is ansible actuator .remoteMachineSet For multi cluster management , Synchronize metadata cluster and business cluster machine state . There are others, such as docking with the public cloud 、dns、lb Wait for the controller .

3.3 Kubernetes-Operator  framework

vivo The application of is distributed in multiple data centers K8s On the cluster , Provides centralized multi cloud management 、 Unified scheduling 、 High availability 、 Failure recovery and other key features . It mainly builds a metadata cluster pass The platform manages multiple businesses K8s colony . Among the many key components , among kubernetes-operator It is deployed in the metadata cluster , It also runs independently machine The controller manages the physical resources .

The following is an example of some scenarios ：

Scene one ：

When a large number of applications migrate to kubernets On , The administrator evaluates that the cluster needs to be expanded . First, you need to approve the physical resources and pass pass The platform generates the corresponding machine Of CR resources , At this time, the physical machine is in the standby pool ,machine CR Is idle . When the administrator creates ClusterDeploment To which MachineSet The idle state will be associated machine, Get free machine resources , We can observe the current need to operate the machine IP Address generation corresponds to inventory And variables , And create configmap And mount to job. Perform capacity expansion ansible Script , If job After successful execution, you will update machine The status of is deployed. Simultaneously synchronize across clusters node The controller of will check the current capacity expansion node Is it ready, If ready, It will update the current one machine by Ready state , To complete the entire capacity expansion process .

Scene two ：

When one of the business clusters fails , Unable to provide services , Trigger the recovery process , Unified resource scheduling . At the same time, the business strategy is to allocate to multiple business clusters , A standby cluster is also configured , No instance is allocated on standby cluster , The standby cluster does not actually exist .

There are the following 2 In this case ：

• Other service clusters can carry the services of the fault cluster ,kubernetes-operator No action is required .

• If other service clusters cannot carry the services of the failed cluster . The container platform starts estimating resources , call kubernetes-operator Create clusters , creating clusterDeployment Select a physical machine from the standby pool , Observe the current need to operate the machine IP Address generation corresponds to inventory And variables , establish configmap And mount to job. Perform cluster installation ansible Script ,  After the normal deployment of the cluster, start the business migration .

3.4 Kubernetes-Operator  Execute the process

• The cluster administrator or container platform triggers the creation ClusterDeployment Of CR, To define the operation of the current cluster .

• ClusterDeployment The controller senses the change and enters the controller .

• Start to create machineSet And correlation machine  resources .

• ClusterInstall  Controller awareness ClusterDeployment and Machineset The change of , Start Statistics machine resources , establish configmap and job, Parameter specifies the of the operation ansible yml entrance , Perform expansion and contraction 、 Upgrade and installation .

• The scheduler senses job Created pod resources , To schedule .

• The scheduler calls K8s Client update pod Of binding resources .

• kubelet Perceive pod Scheduling results of , establish pod Start execution ansible playbook.

• job controller perception job Implementation status of , to update ClusterDeployment state . Under the general strategy job controller Will clean up configmap and job resources .

• NodeHealthy perception K8s Of node Is it ready, Synchronization machine The state of .

• addons  The controller senses whether the cluster is ready, If ready To perform relevant addons Installation and upgrade of plug-ins .

Four 、 summary

vivo On a large scale K8s In the practice of cluster operation and maintenance , From the optimization of the underlying cluster deployment tools , To a large number of CI The matrix test ensures the security and stability of our online cluster operation and maintenance . Adopted K8s trusteeship K8s The way comes from the automated management cluster （K8s as a service）, When operator Detect the current cluster status , Judge whether it is consistent with the goal , When there is a discrepancy ,operator Will initiate a specific operation process , Drive the entire cluster to the target state .

At present vivo Its applications are mainly distributed in multiple self built data centers K8s In the cluster , With the continuous growth of applications and complex business scenarios , You need to provide multiple services that span self built computer rooms and clouds K8s The cluster runs the original cloud generated applications . Need Kubernetes-Operator Provide docking with public cloud infrastructure 、apiserver Load balancing of 、 The Internet 、dns and Cloud Provider  etc. . It needs continuous improvement , Reduce K8s The difficulty of cluster operation and maintenance .