The last line of defense of cloud primary mixing department: node waterline design

author ： Nanyi

introduction

In Ali group , In offline mixing technology from  2014  Year begins , After seven years of double eleven test , The interior has been promoted on a large scale , Save billions of resource costs for Alibaba group every year , The overall resource utilization rate reached  70%  about , Achieve industry-leading . These two years , We began to export the mixed technology of the group to the industry through product , It is seamlessly installed on the standard native platform through plug-in k8s On the cluster , Cooperate with the management and control and operation and maintenance capacity of the mixing department , Improve the resource utilization of the cluster and the comprehensive user experience of the products .

As the mixed department is a complex technology and operation and maintenance system , Include  K8s  Dispatch 、OS  Isolation 、 Observability and so on , Previous  2  An article ：

after  7  Annual double  11  actual combat , How does Alibaba define the scheduling priority and service quality of the cloud native hybrid department ？

How to efficiently allocate cluster resources by using resource quotas in the cloud native hybrid scenario ？

Today we want to pay attention to the last line of defense of the mixed department when it is running on a single machine —— Single machine water level design .

Why do I need a stand-alone water mark

As shown in the figure above ,Pod  Three life cycle phases of the runtime , After quota inspection and scheduling , finally , Different  Qos  Levels of  Pod  Running on the same node , This is the time , High excellent and medium excellent  Pod  The total container allocation on the node is used , Low and excellent  Pod, It is based on the actual resource consumption of high school students , Then it is scheduled by the scheduler to run on the node . You can see it in the picture below , When there are more free resources on a node , It can be used for low-quality resources , And when high / Zhongyou  Pod  After the actual resource consumption is higher than a certain value , When the competition for resources is very fierce , Run lower and better on the node  Pod  It will only lead to high / Zhongyou  Pod  The quality of service is impaired , So at this point , We will no longer allow low priority  Pod  Run on this node .

In order to identify or judge the fierce competition of node resources , Then a very logical design is , Check whether the resource utilization on this node is too high .

  If it exceeds a certain utilization rate , Then we need to deal with low priority  Pod  Do the corresponding operation . The critical threshold of this judgment , It's the water level of a single machine .

Another thing you can see here is , The water level is only low and excellent  Pod  Set up , high / Zhongyou  Pod  The water level is not perceived , They can freely use all the resources of the whole machine , All the system behavior is the same as not opening the mixing department .

Classification of water level

For a node whose resources tend to be saturated , We are for low and excellent  Pod  There can be various means of operation , If it's just a simple way to kill low priority  Pod  Words , The whole hybrid system can also work , This is what we call “ deportation ”.

But if after a certain time , If the resource competition on the machine decreases , So low and excellent  Pod  Be killed and restart on another machine , This will greatly extend the low priority  Pod  The execution time of a single task , So when designing the water level of a single machine , We need to make low and excellent as much as possible  Pod  We should also be able to “ Downgrade ” function . therefore , We have a low priority  Pod  Of the  2  Kind of operation , Is to reduce its  CPU  Supply , We call this operation “ suppress ”.

meanwhile , If the resources on a node tend to be saturated , In addition, the more logical system behavior is not to let the new low optimization  Pod  Be dispatched .

So we have low optimization on nodes  Pod  The behavior of  3  Kind of ： suppress 、 Expulsion and prohibition of dispatch , Thus, there are three water levels , meanwhile , about  CPU  Such compressible resources and uncompressed resources such as memory , There are differences in behavior .

notes ： Compressible resources ( for example  CPU  loop ,disk I/O  bandwidth ) It's all speed and recyclable , For one  task  It can reduce the amount of these resources without killing  task; And uncompressed resources ( For example, memory 、 Hard disk space ) These generally don't kill  task  You can't recycle it . From the article 《 stay  Google  Use  Borg  Manage large-scale clusters  5-6》- 6.2  Performance isolation  

[

 

1]

The general list of these water marks is as follows ：

The reasonable configuration value of these water levels , Should be   deportation > suppress > No scheduling .

However, in the actual mixed production , We usually use the same configuration value for the prohibit dispatching water line and the suppress water line , To reduce the understanding cost of system operation and maintenance students , And configuration workload . In this way, it will exist after the merger  CPU  Of  2  Water line , A watermark in memory .

Conditions of expulsion ： Expulsion mode based on satisfaction

This figure shows an example of the actual system operation on a single machine

• stay  t1  Time , When the total resource utilization reaches the suppressed water level , Suppress low priority tasks , Ensure that the overall resource utilization is below the suppressed water level , At this time, low priority tasks will no longer be scheduled

• stay  t3  Time , The overall resource utilization rate began to rise further , When the expulsion water level is reached , Low priority tasks will be deleted and expelled , Ensure high / Medium and excellent resource utilization

An easy design to consider is , Set a delay time before expelling low priority tasks , In this way, low priority  Pod  There are more opportunities to wait until the system has enough resources , Continue operation , However, this design , It will cause several problems ：

• Memory eviction must be real-time , Because there is not enough memory on the node , Can lead to high / There is insufficient memory for medium and excellent tasks  OOM

• This delay time is not easy to configure , Short matching has no effect , Long matching will lead to low quality  Pod  long-term “ hunger ” And cause low quality  Pod  It takes longer to run

• If on a node , There are many low advantages  Pod  It's all running , Whether to expel all the low advantages  Pod？ Is it possible to expel as little as possible  Pod？

therefore , We invented low priority based on satisfaction  Pod  Of  CPU  Resource expulsion mode , The following concepts are defined ：

• Window period ： obtain  CPU  Time window of utilization （ for example  5  minute ）, Average window time  CPU  Utilization rate exceeds the expulsion water level , Begin to expel , Can avoid jitter

• Low excellent  Pod  Resource satisfaction rate ：=  Low excellent  Pod  Actual resource usage / Low excellent  Pod Request  resource

• Low excellent  Pod  Lower limit of satisfaction rate ： A percentage value , Those below this value are considered low and excellent  Pod  The supply of resources is insufficient

such , Low excellent  Pod  The conditions for expulsion become ：

• During the window period ： Average low excellent  Pod  Resource satisfaction rate  <  Low excellent  Pod  Lower limit of satisfaction rate

• During the window period ： Low excellent  Pod  Average  CPU  The utilization rate is close to  100%（ Such as  90%  perhaps  80%）

• current time ： Average low excellent  Pod  Resource satisfaction rate  <  Low excellent  Pod  Lower limit of satisfaction rate

• The latest time ：BE CPU &nbsp; The utilization rate is close to 100%（ Such as  90%  perhaps  80%）

And expel low priority  Pod  The order of is ：

• Priority expulsion scheduling priority  Priority  Low  Pod（ Yes , Even low and excellent  Pod, We can still subdivide different scheduling priorities according to numerical values ）

• If  2  individual  Pod  Scheduling priority is consistent , Then calculate which one to expel  Pod  It will release more resources , Priority expulsion can release more resources

Memory expulsion mode and  CPU  similar , But there is no satisfaction rate , When it comes to the expulsion watermark, expulsion is carried out according to priority and memory size .

notes ： Low excellent  Pod  Rebuild on other nodes , Or rely on low and excellent  Pod  Control system （ for example , Framework for offline computing  Spark/Flink  etc. ）, This reconstruction process often involves the migration and consistency verification of temporarily cached files or data , This reconstruction operation is not suitable for  K8s  The layer takes the initiative to do the operation , Instead, it is handed over to the upper control system or  operator  More appropriate .

expectation ： Is there a better design ？

At the beginning of this article , It refers to measuring the fierce competition of system resources , The simplest and most intuitive thing is to look at resource utilization . Of course , In the actual large-scale cluster operation , We also see that there is not a complete one-to-one correspondence between high resource utilization and fierce resource competition , Some even apply to  CPU  When the utilization rate is very high , Still running stably , Other applications , In a low  CPU  In case of utilization , It will be very “ card ”.

This means that if we have new 、 Better indicators to measure the utilization of the system , So we are right corresponding  Workload  Can have more “ micro ” The operation of , In ensuring application  SLO  At the same time , Improve the resource utilization of the cluster .

Introduction to relevant solutions

Into the  2022  year , The mixed Department has become a very mature technology in Ali , Save Ali billions of costs every year , It is the basic capability of Alibaba data center . Alibaba cloud has also taken these mature technologies for two years , Precipitate into mixed products , Began to serve all walks of life . The relevant capabilities of the cloud native hybrid department have applied for a number of independent intellectual property rights .

In Alibaba cloud's product family , We'll pass the mixed department's ability through  ACK  The agile version , as well as  CNStack(CloudNative Stack) Product family , External disclosure , Combined with the dragon lizard operating system （OpenAnolis）, Form a complete integrated solution for the mixing Department of cloud native data center , Output to our customers .

Reference documents ：

[1]《 stay Google Use Borg Manage large-scale clusters  5-6》：

 

https://my.oschina.net/HardySimpson/blog/517283

Click on

here

, Visit the overall solution of cloud native hybrid department immediately  ACK  The agile version ！