Kubernetes pod troubleshooting guide

Pod misarrangement

---

Pod Various abnormal phenomena , Possible causes and solutions .

The common names used in the troubleshooting process are as follows :

• see Pod state : kubectl get pod <pod-name> -o wide
• see Pod Of yaml To configure : kubectl get pod <pod-name> -o yaml
• see Pod event : kubectl describe pod <pod-name>
• View container log : kubectl logs <pod-name> [-c <container-name>]

Pod There are many states , Here's a list of :

Error : Pod An error occurred during startup

NodeLost : Pod Node lost connection

Unkown : Pod The node is disconnected or other unknown exceptions

Waiting : Pod Waiting to start

Pending : Pod Waiting to be scheduled

ContainerCreating : Pod The container is being created

Terminating : Pod Being destroyed

CrashLoopBackOff ： Container exit ,kubelet It's being rebooted

InvalidImageName ： Unable to resolve image name

ImageInspectError ： Can't verify image

ErrImageNeverPull ： The policy forbids pulling the image

ImagePullBackOff ： Retrying pull

RegistryUnavailable ： Not connected to the mirror Center

ErrImagePull ： General pull image error

CreateContainerConfigError ： Can't create kubelet Container configuration used

CreateContainerError ： Failed to create container

RunContainerError ： Failed to start container

PreStartHookError : perform preStart hook Report errors

PostStartHookError ： perform postStart hook Report errors

ContainersNotInitialized ： The container is not initialized

ContainersNotReady ： The container is not ready

ContainerCreating ： Container creation in progress

PodInitializing ：pod In the initialization

DockerDaemonNotReady ：docker It's not fully started yet

NetworkPluginNotReady ： The network plug-in has not been fully started

 

 

Failed

---

Failure （Failed）：Pod All containers in have been terminated , And at least one container is terminated for failure . That is, the container is not 0 The state exits or is disabled by the system . （ The container process actively exits ）

If the container process exits voluntarily ( Not killed by external interruption ), The exit status code is usually in 0-128 Between , According to the contract , just The status code is... When exiting normally 0,1-127 It indicates that an exception occurred in the program , I took the initiative to quit , For example, it is detected that the startup parameters and conditions do not meet the requirements , Or occurs during operation panic But no capture processing causes the program to exit . Except maybe it's a business process BUG, There are many other possible reasons , Here we list them one by one .

DNS Unable to resolve

Maybe the program depends on the cluster DNS service , For example, connect to the database at startup , Database usage service Both the name and external domain name are required

want DNS analysis , If the parsing fails, the program will report an error and actively exit . Possible causes of parsing failure :

• There is a problem with the cluster network ,Pod Unable to connect to the cluster DNS service
• colony DNS Service to hang , Unable to respond to resolution request
• Service Or the domain name and address are configured incorrectly , Itself is an unresolved address

Program configuration error  

• Configuration file format error , The program failed to start parsing configuration and exited with an error
• The configuration content does not conform to the specification , For example, a field in the configuration is required but not filled in , Configuration verification failed , Program error reporting is active sign out

 

 

Pod Always in Unknown state

---

Usually, the node is disconnected , No status reported to apiserver, After reaching the threshold controller-manager Consider that the node is disconnected and set its state to Unknown .

Possible causes :

• The node is under high load and cannot be reported
• The node is down
• The node is shut down
• The Internet is not working

 

 

 

Pod Always in Error state

---

Usually in Error state explain Pod An error occurred during startup . Common causes include ：

• Rely on the ConfigMap、Secret perhaps PV There is no such thing as
• The requested resource exceeds the limit set by the Administrator , Like more than LimitRange etc.
• Violating the security policy of the cluster , For example, in violation of PodSecurityPolicy etc.
• The container does not have permission to operate the resources in the cluster , For example, open RBAC after , Need to be for ServiceAccount Configure role binding