[tke] troubleshooting tips for container problems

1. There is no... In the container environment shell Environmental Science

Sometimes we want to see something inside the container , But there is no container shell execution environment , For example, I want to see coredns In the container /etc/resolv.conf Whether the content of inherits the configuration of the node correctly , The simple operation steps are as follows （ With docker Take the runtime example ）：

Log in to the node where the container is located , perform docker ps Find the coredns Containers ID, And then use cp The command copies the file and you can see it ：

docker ps | grep < Container name > 
docker cp < Containers ID>:/want/to/see/dir .

2. Common troubleshooting commands

1. see kubelet Guard service log （ The runtime log is similar to ）

#  Find out about  daemon  service 
systemctl list-units | grep <DAEMON_SERVICE_NAME> 

#  Show  kubelet  Scroll through the logs 
journalctl  -u kubelet -f

#  export  kubelet  Log to file 
journalctl  -u kubelet > k.log 

#  see  kubelet 2  Log from hours ago to now 
journalctl  -u kubelet  --since "2 hours ago" | less

2. Get the current resource configuration YAML

kubectl get <RESOURCE> <POD_NAME> -n <NAMESPACE> -o yaml 

 Order sample : 
kubectl get pod nginx-xxx -n default -o yaml 
kubectl get deploy nginx -n default -o yaml

3. View the current status description of the resource

kubectl describe <RESOURCE>  <POD_NAME> -n <NAMESPACE>

 Order sample : 
kubectl describe  pod nginx-xxx -n default
kubectl describe  pvc nginx -n default

4. View container log

#  Dynamic refresh view  Pod  After the specified container in  20  Line logs  
kubectl logs <POD_NAME> -c <CONTAINER_NAME> --tail 20 -f  -n <NAMESPACE>

 Order sample : 
kubectl logs nginx-xxx -c nginx --tail 20 -f -n default 

5. Print out the field value specified by the resource （YAML structure )

kubectl get <RESOURCE> -o custom-columns=<ALIAS_NAME_1>:<RESOURCE_KEY_1>,<ALIAS_NAME_2>:<RESOURCE_KEY_2>

 Order sample : 
#  Print... In resources separately  .metadata.name（ Alias  Name）,.status.eniInfos（ Alias  eni）  field value .
kubectl get nec -ocustom-columns=Name:.metadata.name,eni:.status.eniInfos
#  If there is... In the field "." Symbol ,  Need to be used as a whole "" Expand and translate ,  such as "tke.cloud.tencent.com/eni-ip"  It's written in  "tke\.cloud\.tencent\.com/eni-ip"
kubectl get no -o=custom-columns=NAME:.metadata.name,Allocatable_eni-ip:.status.allocatable."tke\.cloud\.tencent\.com/eni-ip"

6. Use kubectl Execute container command

kubectl exec -it <POD_NAME> -c <CONTAINER_NAME> -n <NAMESPACE> -- <COMMAND>

 Order sample : 
kubectl exec -it nginx-xxx -c nginx -n default  -- sh
kubectl exec -it nginx-xxx -c nginx -n default  -- sleep 100000

7. Use kubectl Create the test Pod

kubectl run busybox --image=busybox --overrides='{ "spec": { "nodeName": "<NODE_NAME>" } }' --command -- sleep 100000

 Order sample ：
#  test  Pod  Run on the specified node "10.0.5.3" On 
kubectl run busybox --image=busybox --overrides='{ "spec": { "nodeName": "10.0.5.3" } }' --command -- sleep 100000

8. more kubectl Command usage

kubectl  Commands are used in various ways ：
kubectl --help 

3. Common packet capture commands

tcpdump -i <INTERFACE_NAME> host <HOST> and port <PORT> -nve

 Order sample :
#  Check in  eth0  Interface  ip  by 8.8.8.8, The port number is  53  Non domain name display details package 
tcpdump -i eth0 host 8.8.8.8 and port 53 -nvve 

#  Check in  eth0  Interface  src ip  by 8.8.8.8, dst ip  by  1.1.1.1  Non domain name display details package 
tcpdump -i eth0 src 8.8.8.8 and dst 1.1.1.1 -nvve 

#  Use  Wireshark  Analyze the complete message file  dns.pcap
tcpdump -i eth0 host 8.8.8.8 and port 53 -s 0 -w dns.pcap