Containerd failed to pull private database image (kubelet)

Recently, I am following the tutorial of high tower teacher to build in binary way k8s

kubernetes v1.20.0      I don't use the latest version , Because of internal network problems .

containerd  v1.5.2        I haven't used it before , But yes docker Familiar with

My environment is an intranet environment , Can't connect to the Internet , Even more unable to connect to the Internet ,

Deployed in the Intranet environment Harbor(private registry), Used as a mirror warehouse , In order to http Form to serve , namely insecure registry,ip The address is 192.168.1.8

also 6 Virtual machine nodes , It is divided into 3 Host node (master) and 3 Platform work node (worker), and 192.168.1.1 Is the gateway router address , So from 2 Start ,2-4 by master,5-7 by worker

192.168.1.2/192.168.1.3/192.168.1.4/192.168.1.5/192.168.1.6/192.168.1.7

stay The first 12 step Deploy CoreDNS Part of it containerd Unable to pull the image in the private database .

First of all, I'm in my private library Harbor There is a sandbox image pause:3.2 , I use docker It can be pulled normally , Explain what I mean Harbor There is no problem .docker By /etc/docker/daemon.json  insecure-registries :["192.168.1.8"] Configured .

When I install the tower, the steps of teacher go to ：

$ kubectl get pods -l k8s-app=kube-dns -n kube-system --kubeconfig=admin.kubeconfig

The container is always being created , So through describe Let's look at the specific problems .

After executing the following order , The specific mistake is ：

$ kubectl describe pod coredns-xxxx-xxx  -n kube-system --kubeconfig=admin.kubeconfig

The first question is ： There is kubelet Go to k8s.gcr.io/pause:3.5 Pull the mirror image , No private Kula take image , The sandbox image cannot be downloaded , That is, it cannot run coreDNS Mirror image .

And I'm on the intranet harbor pause The image address is ： 192.168.1.8/kubernetes/pause:3.2

How to solve this problem ？ 

At first, I thought that the mistake was due to kubelet What's the configuration of , So find teacher Gao TA's document again , I saw it again , No trace was found .

And then I found it containerd Official documents of , Found in the default generated /etc/containerd/config.toml It contains to pull sandbox_image Address , in other words containerd tell kubelet Go to gcr.io Pull the mirror image , We modify this address to make kubelet Go to our private library to get the mirror .

$ containerd config default > /etc/containerd/config.toml

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image="k8s.gcr.io/pause:3.5"

Here's the modified configuration , Then restart three worker Node containerd

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image="192.168.1.8/kubernetes/pause:3.2"

$ systemctl restart containerd

The second question is ： I continue to pass kubectl descirbe Command to see the situation , The result of this time is that the address has indeed changed , But I still haven't pulled it down from the private library .

I found a lot of documents , Include github Of issues and discussions Many people have encountered this problem , But there is no detailed solution , Especially after reading this manual https://github.com/containerd/containerd/blob/release/1.5/docs/cri/registry.md, Configured according to the manual for a long time , No effect , It's a bit of a waste of time .

To sum up ： Nothing more than modification config.toml Inside  insecure_skip_verify And in [plugins."io.containerd.grpc.v1.cri".registry] newly added mirrors、auths or configs, But the results are invalid .

also issues People who say ctr Can't use cri Of config, in other words config.toml yes cri Configuration of , Can pass ：

$ ctr images pull --plain-http  192.168.1.8/kubernetes/pause:3.2

To pull , however kubelet I don't know ctr, Because ctr It didn't come true cri Interface , Specific performance: ：

$ ctr images pull --plain-http  192.168.1.8/kubernetes/pause:3.2

$ ctr images ls   // We found this image , But when we do 

$ crictl images   // I didn't find the one just pulled down pause:3.2 Mirror image 

That is, we still can't let kubelet Take it from the private library pause:3.2 The problem of mirroring , Are the following errors ：

Failed CreatePodSandBox 7s(x22 over 4m42s) kubelet Failed to create pod sandbox: rpc error: code=Unknown desc = failed to get sandbox image "192.168.1.8/kubernetes/pause:3.2" :failed to pull image "192.168.1.8/kubernetes/pause:3.2": failed to pull and unpack image "192.168.1.8/kubernetes/pause:3.2": failed to resolve reference "192.168.1.8/kubernetes/pause:3.2": failed to do request: Head "https://192.168.1.8/v2/kubernetes/pause/manifests/3.2": dial tcp 192.168.1.8:443: connect:connection refused 

I looked for the document again , Another day of fierce fighting , Found in containerd： To configure hosts There is one of them. Bypass TLS Verification Example, I immediately matched it according to the tutorial , The problem is solved immediately .

The solution is ：

$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io
    └── hosts.toml

$ cat /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."http://192.168.1.8:80"]
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

There is also the modification of your config.toml Medium 192.168.1.8 by docker.io , Yes , You read that right , Be sure to modify here .

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image="docker.io/kubernetes/pause:3.2"

This is because containerd It's set up hosts, You can understand it as interceptor , Will visit docker.io Replace the link above with our hosts.toml The private library address set inside .

In the future, the image address in all your private libraries is not 192.168.1.8/repo/image:version , It is docker.io/repo/image:version 

That is to say, it is your yaml In the document image Mirror address , Change it to docker.io/repo/image:version  , however harbor Without any changes .

$ systemctl restart containerd

Don't forget to restart all worker node .

Reference link ：

1. High tower teacher ：kubernetes the hard way 

GitHub - kelseyhightower/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.

2.containerd: To configure registry 

https://github.com/containerd/containerd/blob/release/1.5/docs/cri/registry.md 

3.containerd： To configure hosts

 https://github.com/containerd/containerd/blob/release/1.5/docs/hosts.md