Troubleshooting the kubernetes problem: deleting the rancher's namespace by mistake causes the node to be emptied

This article excerpts from kubernetes Learning notes

Problem description

The nodes of the cluster suddenly disappeared (kubectl get node It's empty ), Cause cluster paralysis , But in fact, the virtual machines corresponding to the nodes are still . Because the cluster did not open the audit , So it's not easy to check node What was deleted .

Fast recovery

Because only k8s node Resource deleted , The actual machines are still there , We can restart nodes in batches , Automatic pull up kubelet Re registration node, Can be restored .

Suspicious operation

Found before the node disappeared , There is a suspicious operation : Some students found that in another cluster, there were a lot of messy namespace ( such as c-dxkxf), View these namespace No workload is running in , It could be a test created by someone else namespace, Just delete it .

analysis

Delete namespace Is installed in the cluster of rancher, Suspected of being deleted namespace yes rancher Automatically created .

rancher Managed other k8s colony , Architecture diagram :

guess : Delete the namespace yes rancher Created , Cleaned up when deleting rancher Resources for , It also triggers rancher clear node The logic of .

Analog reproduction

Try to simulate a recurrence , Verify the conjecture :

1. Create a k8s colony , As rancher Of root cluster, And will rancher Install in .
2. Get into rancher web Interface , Create a cluster, Use import The way :

1. Output cluster name:

1. Pop up the prompt , Let the following... Be executed in another cluster kubectl Command to import it into rancher:

1. Create another k8s Cluster as rancher Managed clusters , And will kubeconfig Import local for later use kubectl operation .
2. Import kubeconfig And switch context after , perform rancher Provided kubectl Command to import the cluster into rancher:

You can see in the managed TKE Automatically created in the cluster cattle-system Namespace , And run some rancher Of agent:

1. take context Switch to installation rancher The cluster of (root cluster), You can find that after adding a cluster , Automatically created some namespace: 1 individual c- At the beginning ,2 individual -p At the beginning :

Guess it is c- At the beginning namespace By rancher Used to store the added cluster Information about ;-p Used to store project Relevant information , The official also said that it would be automatically for each cluster establish 2 individual project:

1. See what rancher Of crd, There is one nodes.management.cattle.io Quite conspicuous , Obviously used to store cluster Of node Information :

1. look down node Where to store it namespace ( As expected c- At the beginning namespace in ):

1. Try to delete c- At the beginning namesapce, And switch context To the added cluster , perform kubectl get node:

Nodes are cleared , Problem recurrence .

Conclusion

Experimental proof ,rancher Of c- At the beginning namespace Saved the... Of the added cluster node Information , If you delete this namespace, That is, the stored node Information ,rancher watch The associated cluster will be deleted automatically k8s node resources .

therefore , Never clean up easily rancher Created namespace,rancher Some stateful information is stored directly in root cluster in ( adopt CRD resources ), Delete namespace May cause very serious consequences .