How to configure tke cluster node Max pod

Use tke When the cluster deploys Services , There is often a situation , Namely pod A kind of pending Unable to successfully schedule to node , Check the event prompt and report an error

0/9 nodes are available: 1 node(s) didn't match Pod's node affinity, 8 Too many pods.

Copy

From the log , yes 1 Nodes do not meet node affinity , also 8 Nodes pod Too much , This pod What is the reason for too much ？

Actually tke We will set the maximum capacity of each node of the cluster pod Number , It's usually in kubelet Start configuration of /etc/kubernetes/kubelet Through the file MAX_PODS To define , Generally, it will be synchronized to the node yaml Of status.allocatable.pods Field , When a node's pod The number of max_pods When it's worth , Then schedule to the node pod Will fail , Tips Too many pods.

Then the maximum operation of the node pod How exactly is quantity defined ？ So let's talk about that tke Nodes in different network modes max_pods How to set it .

1. GlobalRouter Mode node maximum pod Number setting

GlobalRouter Network mode is container service TKE Based on the underlying private network VPC Global routing capability , Container network and VPC Routing strategy of mutual visits ,GlobalRouter When a pattern cluster is created, the nodes will be pod Set the upper limit of quantity , Because you need to allocate containers to nodes according to the network configuration cidr.

therefore GlobalRouter Of kubelet Configured max_pods Is the container of a single node cidr ip Number , And then subtract 3 individual , Why should we subtract 3 A? ？ Because there are three addresses that cannot be assigned, which are ： network number 、 Broadcast address and gateway address , therefore Node maximal Pod number = cidr - 3.

For example, if you create a cluster with a single node, set the maximum pod The number is 32, In fact, the node can hold at most 29 individual pod, When the first 30 individual pod If you want to continue running to the node, you will report an error node Too many pods.

Many people have a question here , If my node resources are sufficient , I want to continue scheduling up pod, Is it possible to modify nodes kubelet Parameters are OK ？ In fact, theoretically, this is feasible , In this way, nodes can indeed schedule more pod, But there will be some problems .

If you change the value too large , Will lead to subsequent pod Not assigned ip, Because a node is available pod ip Is constant cidr - 3, however pod The scheduling of will only look at the nodes max-pods Is it full , It does not judge the container network segment ip Is it insufficient , Therefore, the container network segment of the possible node ip Run out of ,pod It will still schedule to the node , Lead to pod Can't assign ip.

The other thing to note is that max-pod Is the containing node hostnetwork Pattern pod,hostnetwork Pattern pod It will not occupy the container network segment ip, But it will count in max-pod In the quantity , Then there is another problem , Although there are nodes cird - 3 And the container ip You can give pod Use , But the actual scheduling is not so much GlobalRouter Pattern pod To the node , Because each node will have some hostnetwork Mode pod, It will take up some pod Number , Therefore, it can actually be allocated to GlobalRouter Pattern pod There will be more than cidr - 3 Less . In fact, it can be appropriately given here kubelet Of pod Increase the number , To hold hostnetwork Pattern pod, Here, the container network segment of each node ip Can make full use of .

2. vpc-cni Mode node maximum pod Number setting

It says GlobalRouter Mode max-pod How is it set up , So let's talk about that vpc-cni How to set in mode ,GlobalRouter Because each node will be assigned a container network segment ,max-pod According to the container network segment cidr To set up , however vpc-cni In mode, the container network segment is vpc subnet ,pod ip Is obtained from the subnet , That of each node max-pods How to set it ？

vpc-cni The model here is more complex ,vpc-cni In mode, there will be a maximum eni ip Quantitative restriction , This is based on the model configuration , It also supports setting single node when creating clusters max-pod, The default is set 64,.vpc-cni The pattern defines the node's max-pods According to the comparison model eni ip Quantity limits and cluster creation settings pod Upper limit size , take 2 The larger of them is set as the node max-pods.

Then the node eni How did the quantity limit come about ？vpc-cni Is the ability to use elastic network cards , Therefore, the restrictions here are also based on the elastic network card , For details, please refer to the document https://cloud.tencent.com/document/product/576/18527

Actually in tke The console also prompts you to select nodes

So how did these quantities come about ？ Let's briefly say , Each model has 2 A limit , One is the number of elastic network cards , There is also the elastic network card intranet ip Quantitative restriction . The elastic network card is limited to eni_num, Elastic network card Intranet ip The number of eni_ip_num

Because the quota of elastic network cards will include the primary network card , Here, the number of elastic network cards used for the container needs to be eni_num-1 Elastic network card Intranet ip Contains the master of the network card ip, Lord ip It cannot be used for containers , So for containers ip by eni_ip_num-1 So finally, a certain model can be used eni ip yes (eni_num-1) * (eni_ip_num-1)

Take the standard model here S5 Of 1 Take nuclear models for example

Finally calculated eni The number of (2 - 1)*(6 - 1)=5, in other words 1C Of S5 Model join tke colony , Model eni The quantity limit is 5 individual . If it is standard S5 Of 8 Nuclear type , that eni The number is (6 - 1)*(20 - 1)=95, in other words 8C Of S5 Model join tke colony , Model eni The quantity limit is 95 individual .

If you create a cluster, the default setting is node node pod The upper limit is 64, Here, when joining the cluster, the accountant calculates the model eni Quantity goes with 64 Compare , Take the larger one as the node max-pods.

in other words 1C Of S5 Model join tke colony , Node max-pod The value is set to 61, Here the back-end design follows GlobalRouter Logic will still subtract 3 individual ip, So it is 61.

8C Of S5 Model join tke colony , Node max-pod The value is set to 95.