Operator-1 first acquaintance with operator

background :

Contact kubernetes For many years , I heard all kinds of things at the beginning Operator Of , But I have never learned more about it Operator
. Start experiencing simple Operator

Operator First experience

What is? Operator?

reference ： Red hat official documents   What is? Kubernetes Operator？
**coreos2016 In introducing ,** It's a package 、 Deployment and management Kubernetes Applied method

1. crd webhook controller

development tool :

what is crd

**CRD ** The full name is Custom Resource Definition, CRD It is an extension of native without coding kubenetes API How to interface . Suitable for expansion kubernetes Custom interfaces and functions of . If you want to add logic more flexibly, you need API Aggregation The way .

Start to prepare

There are several common development tools ：

My development tools Include goland kubebuilder kustomize,kubernetes1.23.6, The work environment rocky linux 8.5 go 1.17
Be careful ： Be sure to have a look go edition The version corresponding to the development tool , As well as kubernetes Version of

kubebuilder kustomize install

 https://github.com/kubernetes-sigs/kubebuilder/releases

Log in and copy

[[email protected] ~]# wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.5.0/kubebuilder_linux_amd64

     

      
1.
     

Log in and copy

[[email protected] ~]# mv kubebuilder_linux_amd64 /usr/bin/kubebuilder
[[email protected] ~]# chmod +x /usr/bin/kubebuilder
[[email protected] ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.5.0", KubernetesVendor:"1.24.1", GitCommit:"26d12ab1134964dbbc3f68877ebe9cf6314e926a", BuildDate:"2022-06-24T12:17:52Z", GoOs:"linux", GoArch:"amd64"}

     

      
1.
      
2.
      
3.
      
4.
     

Log in and copy

[email protected] ~]# wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.5.5/kustomize_v4.5.5_linux_amd64.tar.gz


     

      
1.
      
2.
     

Log in and copy

[[email protected] ~]# tar zxvf kustomize_v4.5.5_linux_amd64.tar.gz 
kustomize
[[email protected] ~]# chmod +x kustomize
[[email protected] ~]# mv kustomize /usr/bin/kustomize
[[email protected] ~]# kustomize version
{Version:kustomize/v4.5.5 GitCommit:daa3e5e2c2d3a4b8c94021a7384bfb06734bcd26 BuildDate:2022-05-20T20:25:40Z GoOs:linux GoArch:amd64}

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

Create and initialize project

goland Create a kube-oprator1 Project ：

The terminal executes a command ：

Log in and copy

[[email protected] kube-oprator1]$ kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"


     

      
1.
      
2.
     

It seems to remind me go Version is too low ？（go edition 1.17.6 My is ）

Upgrade go edition

Be careful ： Not necessary , The latter is lowered kubebuilder Version of .go Keep the version 1.17 Version of the ,

Browser open  https://golang.google.cn/dl/ go The download page , choice 1.17 Download the latest version and replace the local GO edition ！

Log in and copy

[[email protected] ~]# wget https://golang.google.cn/dl/go1.17.11.linux-amd64.tar.gz
[[email protected] ~]# tar zxvf go1.17.11.linux-amd64.tar.gz
[[email protected] ~]# which go
/usr/go/bin/go
[[email protected] ~]# cd go/
[[email protected] ~]# cp -Ra * /usr/go/
[[email protected] go]# go version go1.17.11 linux/amd64

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
     

.... It seems that the report is wrong , Took a closer look  https://github.com/kubernetes-sigs/kubebuilder/releases I'd better change it kubebuilder Version …

kubebuilder edition 3.4.1

Log in and copy

[[email protected] ~]# wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.4.1/kubebuilder_linux_amd64
[[email protected] ~]# mv kubebuilder_linux_amd64 /usr/bin/kubebuilder
mv： Is it covered? '/usr/bin/kubebuilder'？ y
[[email protected] ~]# chmod +x /usr/bin/kubebuilder 
[[email protected] ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.4.1", KubernetesVendor:"1.23.5", GitCommit:"d59d7882ce95ce5de10238e135ddff31d8ede026", BuildDate:"2022-05-06T13:58:56Z", GoOs:"linux", GoArch:"amd64"}

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

Log in and copy

kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"

     

      
1.
     

The structure of the generated directory is as follows ：

Focus on config/default/ kustomization.yaml file ：

Now you can understand the configuration , Namespace ！ I will not change the default configuration here ！

Log in and copy

 [[email protected] kube-oprator1]$ kubebuilder create api --group myapp1 --version v1 --kind Redis


     

      
1.
      
2.
     

The directory structure is as follows

Be careful : About domain group version kind Corresponding :

Log in and copy

apiVersion:myapp1.zhangpeng.com/v1
kind: Redis

     

      
1.
      
2.
     

Simply create one crd

api/v1/redis_type.go


Show me how to delete Foo Field , Add one Port Field , Set up port Field is int type ！

With test Under the table of contents yaml File customization crd

test/redis.yaml

Log in and copy

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1011

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

make install establish crd

Log in and copy

[[email protected] kube-oprator1]$ kubectl get crd
No resources found
[[email protected] kube-oprator1]$ make install 
GOBIN=/home/zhangpeng/GolandProjects/kube-oprator1/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
/home/zhangpeng/GolandProjects/kube-oprator1/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/zhangpeng/GolandProjects/kube-oprator1/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp1.zhangpeng.com created
[[email protected] kube-oprator1]$ kubectl get crd
NAME                         CREATED AT
redis.myapp1.zhangpeng.com   2022-06-28T06:44:52Z


     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
     

About reconcile

controllers/redis_controller.go
About reconcile Just don't ask for a better understanding

Log in and copy

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	redis := &myapp1v1.Redis{}
	if err := r.Get(ctx, req.NamespacedName, redis); err != nil {
		fmt.Println(err)
	} else {
		fmt.Println("object", redis)
	}

	return ctrl.Result{}, nil
}

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
      
11.
      
12.
      
13.
     

Local debugging make run

As soon as the terminal is running

Log in and copy

[[email protected] kube-oprator1]$ maker run  

     

      
1.
     

terminal 2 function

Log in and copy

 
 [[email protected] kube-oprator1]$ kubectl apply -f test/redis.yaml

     

      
1.
      
2.
     

Observation terminal 1 The output is as follows ：

Preliminary release to kubernetes colony

notes ： My environment has installed podman, About podman To baidu , The mirror warehouse uses Tencent cloud image warehouse Personal Edition

About Podman

First modify docker The build command is podman!

podman login Remember the password … Basic heel docker You can use it in the same way

Log in and copy

[[email protected] kube-oprator1]$ podman login --username=xxxxx ccr.ccs.tencentyun.com


     

      
1.
      
2.
     

dockerhub Speed up

In particular dockerhub Speed up

Log in and copy

[[email protected] kube-oprator1]$  vim /etc/containers/registries.conf

     

      
1.
     

The acceleration address is added at the end of the file ！

Log in and copy

short-name-mode = "permissive"
[[registry]]
prefix="docker.io"
location="pvurwzu6.mirror.aliyuncs.com"

     

      
1.
      
2.
      
3.
      
4.
     

restart podman service

Log in and copy

[[email protected] kube-oprator1]$  systemctl restart podman

     

      
1.
     

Build publishing image

Dockerfile Add... To the file GOPROXY

Log in and copy

ENV GOPROXY=https://goproxy.io

     

      
1.
     

Log in and copy

[[email protected] kube-oprator1]$ make docker-build docker-push IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

     

      
1.
     

Incidentally, I found that one of Tencent cloud's personal warehouses is not displayed OCI-Image The size of bug…

notes ： The process is tortuous . In the middle, there is science that doesn't move under the mirror on the Internet , Autonomic brain . Such as “gcr.io/distroless/static:nonroot Mirror my operating environment as rocky linux 8.5 When I couldn't download it, I went online directly …

How to publish ：

Log in and copy

[[email protected] kube-oprator1]$ make deploy IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

     

      
1.
     

make Failed again, finally according to Makefile in deploy Manually execute the following commands ：

Log in and copy

[[email protected] kube-oprator1]$ cd config/manager &&  kustomize edit set image controller=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ kustomize build config/default | kubectl apply -f -


     

      
1.
      
2.
      
3.
     

Be careful ： Both commands are in kube-oprator1 Executed under the project root directory ！


The default namespace is not modified kube-oprator1-system namespace Under the namespace pod state ！

Log in and copy

[[email protected] kube-oprator1]$ kubectl get ns
NAME                   STATUS   AGE
default                Active   61d
kube-node-lease        Active   61d
kube-oprator1-system   Active   25h
kube-public            Active   61d
kube-system            Active   61d
zhangpeng1             Active   8d
[[email protected] kube-oprator1]$ kubectl get pods -n kube-oprator1-system


     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
      
10.
     

Theoretically pod The deployment was not successful , Here's why ：gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0 Unable to download … I use benzene method ,rocky Development machine science downloads images online and uploads them to Tencent and image warehouse , then pull Mirror to kuberntes machine work node . Yes, of course work Node i only have one test environment, which is OK ～

wait for pod running

CRD Custom resource simple validation

With test/redis.yaml For example

Log in and copy

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1011

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

reference  https://book.kubebuilder.io/reference/markers/crd-validation.html Just set it up port The scope of the ！

Log in and copy

	// +kubebuilder:validation:Minimum:=1024
	// +kubebuilder:validation:Maximum:=10240

     

      
1.
      
2.
     

make install Still failed ！ Or manual command
notes ： I made mistakes in this place several times , The reason is that I thought make install =kustomize build config/crd | kubectl apply -f -, Took a closer look Makefile：

contain manifests Steps for , I tried it, and it worked ！

Log in and copy

[[email protected] kube-oprator1]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[[email protected] kube-oprator1]$ kustomize build config/crd | kubectl apply -f -

     

      
1.
      
2.
      
3.
     

Log in and copy

[[email protected] kube-oprator1]$ kubectl get crd redis.myapp1.zhangpeng.com -o yaml


     

      
1.
      
2.
     

Log in and copy

[[email protected] kube-oprator1]$ kubectl delete -f test/redis.yaml
[[email protected] kube-oprator1]$ kubectl apply -f test/redis.yaml

     

      
1.
      
2.
     

The output is as follows ： The port is less than 1024 Cannot create successfully

modify test/redis.yaml port: 1024

Log in and copy

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1024

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

Log in and copy

[[email protected] cert]$ kubectl get pods -A|grep cert
cert-manager           cert-manager-677874db78-zcm6l                       1/1     Running            0                  14m
cert-manager           cert-manager-cainjector-6c5bf7b759-mf4gf            1/1     Running            0                  14m
cert-manager           cert-manager-webhook-5685fdbc4b-ncrxl               1/1     Running            0                  14m


     

      
1.
      
2.
      
3.
      
4.
      
5.
     

webhook A simple test

Simple access controller webhook create

Log in and copy

[[email protected] kube-oprator1]$ kubebuilder create webhook --group myapp1 --version v1 --kind Redis --defaulting --programmatic-validation


     

      
1.
      
2.
     

kube-oprator1 api/v1 Added... To the directory webhook Related documents of , Made a simple verification ** name=zhangpeng**

Log in and copy

func (r *Redis) ValidateCreate() error {
	redislog.Info("validate create", "name", r.Name)
	if r.Name == "zhangpeng" {
		return errors.New("error name")
	}

	// TODO(user): fill in your validation logic upon object creation.
	return nil
}

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
      
7.
      
8.
      
9.
     

Certificate management cert-manager:

visit  https://github.com/cert-manager/cert-manager/releases The download page ,1.19.0 yes alpha edition I used it 1.18.2 Version of ！

Log in and copy

[[email protected] cert]$ pwd
/home/zhangpeng/cert
[[email protected] cert]$ wget https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
[[email protected] cert]$ kubectl apply -f cert-manager.yaml
[[email protected] cert]$ kubectl get pods -A|grep cert

     

      
1.
      
2.
      
3.
      
4.
      
5.
     

Be careful ： Mirror images still need to be downloaded scientifically

Modify file

config/default/kustomization.yaml The down arrow callout section unlocks the annotation


**config/manager/manager.yaml **

Delete crd

make uninstall Yes, but mine make Always fail … Delete... Directly ！

Log in and copy

[[email protected] kube-oprator1]$kubectl delete crd redis.myapp1.zhangpeng.com


     

      
1.
      
2.
     

Packaging image publishing

Packaging and publishing images , In fact, it is better to modify an image label tag, Here is a demonstration That's it first ！make install I don't know there is a problem Direct copy Makefile The order in ！ Build an image and publish it ！

Log in and copy

[[email protected] kube-oprator1]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[[email protected] kube-oprator1]$ kustomize build config/crd  | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp1.zhangpeng.com configured

[[email protected] kube-oprator1]$ make docker-build docker-push IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ make deploy IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

     

      
1.
      
2.
      
3.
      
4.
      
5.
      
6.
     

Log in and copy

[[email protected] kube-oprator1]$ cd config/manager && kustomize edit set image controller=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ kustomize build config/default | kubectl apply -f -

     

      
1.
      
2.
     

EN, modified to zhangpeng1

Created successfully ？ to glance at make run,but make run Can't run ？

The local debug mode is commented out main.go SetupWebhookWithManager

summary ：

1. Pay attention to the version matching between development tools
2.make Look at it when you fail Makefile You can manually run the relevant commands in
3. Resource cleanup , Local debugging mode
4. Next, we are going to design a simple Operator?