Sailing with karmada: multi cluster management of massive nodes

stay KubeCon2021 On , Shenyifan, the cloud platform architect of ICBC software development center, and wangzefeng, the person in charge of Huawei cloud native open source, made 《 And Karmada Sail together ： Multi cluster management of massive nodes 》 Keynote speech , Shared

Industrial and Commercial Bank of China k8s Cluster management practice process .

*Karmada The project introduction is shared by wangzefeng , The rest is shared by shenyifan

Construction status of ICBC cloud platform

• The current business cloud scenarios of ICBC are rich and diverse , There are core business applications represented by payment lines such as Spring Festival red envelopes , There is also MySQL、Redis As a representative of the technical support applications , It also includes blockchain 、 Artificial intelligence and other new technology fields .

• At present, ICBC cloud platform has also conducted in-depth customized research and development based on mainstream open source projects , It ensures the overall autonomy and controllability .

• From the perspective of construction , It is also the largest container cloud in the industry , At present, the number has reached 28 ten thousand +.

Typical business requirements and cloud native infrastructure status

In the context of large-scale cloud native infrastructure , The specific requirements of ICBC's business on us and the current situation ：

Typical business requirements mainly include ： The business needs highly available deployment 、 Applications need to scale and schedule across clusters , Some business products need to be right K8s Version has a specific dependency .

Based on the above situation , The current status of ICBC's cloud native infrastructure is as follows ：

• A single cluster requires high reliability . The number of nodes in our overall single cluster is 2000 following , This is to narrow down the impact of cluster failure rate .

• The resource pool grows rapidly with the business . At present, the new business application is fully on the cloud , Stock applications are constantly migrating to the cloud , Now all core applications have been put into the cloud .

• Business level heterogeneous cluster . Business to specific K8s The version depends on , And there are a lot of heterogeneous CNI、CSI、 Including some underlying hardware heterogeneity .

• Multifocal 、 Multicenter 、 The current situation of cloudy construction , ICBC's business includes the general bank cloud 、 Branch cloud , Ecological cloud, etc . In terms of fault domain, the data center construction of two places and three centers , And there are more fine-grained divisions of multiple fault domains within the data center .

Key challenges

Based on the above situation , Within ICBC K8s The number of clusters has actually reached 100 More than , At present, the container cloud platform is used for unified management . But in the process of sustainable development , We also face the following problems ：

• Limited availability , because K8s The cluster itself is also a fault domain , At present, there is no automatic recovery across fault domains .

• Limited resources , The overall application scheduling and elastic scaling are limited by a single cluster .

• The cluster is opaque ： The clusters are now heterogeneous 、 Fault domain and other attributes , Business teams need to be aware of the underlying clusters to make their own choices K8s colony , That led to it. K8s The cluster itself is opaque to upper layer applications .

• Repeat configuration . Although our business is configured and entered on the cloud management platform , But the specific configuration needs to be distributed to each cluster , And each cluster shall ensure synchronization .

Design objectives

Facing these challenges , We have drawn up design goals for the management of multiple clusters ：

• In the multi cluster management plane ： Cluster management and overall life cycle management of the cluster , And has a unified standard API entrance .

• In terms of resource management , You need to support multiple versions of comprehensive K8s resources , It also requires multi-dimensional resources Override Support .

• In the aspect of cross cluster automatic scheduling , Scheduling needs to be based on fault domain 、 Automatic scheduling of resource allowance, etc , And it can automatically scale across clusters .

• Disaster recovery , Automatic recovery of cross cluster resources is required , And the management plane and the business cluster need to be decoupled .

• Compatibility , For heterogeneous clusters with a large number of stocks, smooth management is required , At the same time, the project itself needs high expansibility and community activity .

Joint innovation

With clear design objectives , The next step is how to implement the landing . First, for commercial products , Considering the binding of a single manufacturer and the fact that it does not meet the requirements of independent control , First we pass, And research Kubefed after , Found that it uses non-native API, It is too difficult for our stock cluster to migrate , And the community activity has decreased recently . Finally, we choose the way that best suits our needs , Based on their own joint research and development , At the same time, ICBC financial cloud itself is also based on open source , So we are also actively engaged in open source co construction , Promote community development in this virtuous circle , Finally, choose to jointly initiate karmada project .

Karmada  project

Why not in the original KubeFed Continue to evolve on the basis of , But to launch a new project ？

In fact, what we consider at the beginning of the project is Kubernetes Federation Of V3 edition , And quickly completed the prototype development . However, in the later communication with many community users , We found that in a narrow sense Federation The scope of the project does not completely cover the expected capability map . except Federation In addition to the load management capability of multi cluster applications included in itself , In fact, we also hope to provide resource scheduling like multi cluster 、 Fault migration 、 Automatic telescopic , And service discovery like multi cluster 、 Data automation and cluster lifecycle management supported by multiple platforms , Provide users with open source software stations that are multi cloud and multi cluster out of the box . So a trust in  CNCF Neutral open source project , It will be more suitable for long-term technological evolution and community development .

Karmada Technical panorama

Karmada The core structure of

stay Karmada The core architecture of , It has learned from the experience and experience of many community sponsors in multi cluster management , We focused on K8s Native API Support 、 And framework extensibility .

And K8s Single set group frame structure is slightly similar ：

• Karmada  The control surface has its own independent APIserver, To provide K8s Native API as well as Karmada An extension of API

• adopt Karmada Scheduler, Provide targeted  [ Fault domain 、 Cluster resources 、K8s Version and cluster startup plug-ins 、 Multi weighted ] Scheduling policy support , And it is convenient for users to customize the extension

• And member The synchronization aspect of the cluster ,Karmada Agent Of pull Working mode , It can effectively disperse and control the surface pressure , Realize the management of super large-scale multi cluster resource pool .

• meanwhile ,Karmada And support ExecutionController As well as the integration KubeEdge Method to implement the public cloud 、 Private cloud, edge and other network environments K8s Direct management of the cluster .

• And with the help of ExecutionSpace The design of the ,Karmada It realizes the access rights and resource isolation between different clusters , To meet the security needs in multi cluster scenarios .

Karmada  The core concept

stay Karmada The design aspects of the core concepts , We define all workload related resource objects entered by the user as Resource Template, This is strict K8s Native API, Support includes deployment,service,configmap etc. , And user-defined CRD Resource objects . such , The user does not need any modification , You can directly use the original single cluster YAML perhaps API To create multi cluster applications , And originally K8s The secondary development business platform does not need to be modified .

Split and schedule applications across multiple clusters , Extended Propagation Policy Support definitions that are reused by multiple applications . Thanks to this decoupling design , In the scenario of setting up platform team and business team independently like ICBC , The platform team can set policies for the common high availability deployment model , The business team can continue to use K8s Native single cluster API To manage daily business launch and version upgrade .

Case study ： How to use Karmada Management business

Zero transformation ： Use native API Deploy a 3AZ Highly available applications

So in this example we can see , In fact, the first one is Propagation policy. In this Propagation policy The definition of , The platform team set up  Resource selector, Limit all Deployment, If it has a special Label,HA mode by Multi zone replication, These applications are strictly distributed to three Zone Go inside . On the right is a standard that we are familiar with  Deployment Of  API The definition of . By combining these two definitions , In fact, we can see that the platform team focuses on the setting of the general application deployment model , The business team focuses on the internal of its own applications, such as Image、 edition , And things like this  Container port And so on . and Karmada Be responsible for combining the needs of the two teams , Realize cross region high availability of applications . In the event of any cluster failure , Dynamic scheduling capability can be used , Automatically supplement the application instances of missing clusters or missing zones , To achieve cross cluster failover .

Practical experience

stay Karmada Design R & D practice is in the process of positive cycle of redesign R & D , ICBC has also summed up some of its advantages , It can also be said to be experience . It is mainly divided into the following four categories ： Resource scheduling 、 disaster 、 Cluster management and resource management . I think the following three points deserve special attention , Then it is particularly prominent in the process of real landing .

• Support binding and scheduling of multiple resources
  , This ensures that the business node needs k8s Resources can be scheduled at the same time , It also greatly improves the timeliness of our resource distribution .

• Support k8s Native object
  , This ensures that we have a large number of current k8s External clients require little modification .

• Karmada At present, we support Pull and Push Mode distribution , Adapted to a variety of scenarios
  . Especially in a scenario where we have a large number of clusters , Use Pull Mode can greatly reduce Karmada Control plane performance pressure .

The follow-up plan

stay

Large scale production is implemented

aspect , We hope that the container cloud platform will be a user oriented platform in the future , Bottom based Karmada Uniformly manage the resources and scheduling of multiple clusters , This is used to manage the stock, including heterogeneous clusters 100+ Of k8s colony .

stay

Community contribution

aspect , We want to continue making community contributions . The main features of concern include the smooth migration of stock applications , Can be automatically incorporated into Karmada Federalization . Scaling across clusters 、 Application migration and data linkage , Continue to optimize and implement .

Last , Welcome to Karmada Project GitHub, see Release note And community documentation , Learn more about features and details . If you are using Karmada Any suggestions and feedback in the process of , Welcome to join the community group to communicate and discuss with us ！

attach ：Karmada Community technology exchange address

 

Project address ：

https://github.com/karmada-io/karmada

Slack Address ：

https://karmada-io.slack.com