Oprator-1 first acquaintance with oprator

background :

Contact kubernetes For many years , I heard all kinds of things at the beginning oprator Of , But I have never learned more about it oprator. Start experiencing simple oprator

OPrator First experience

What is? Oprator?

reference ： Red hat official documents What is? Kubernetes Operator？

coreos2016 In introducing , It's a package 、 Deployment and management Kubernetes Applied method

1. crd webhook controller

development tool :

what is crd

CRD The full name is Custom Resource Definition, CRD It is an extension of native without coding kubenetes API How to interface . Suitable for expansion kubernetes Custom interfaces and functions of . If you want to add logic more flexibly, you need API Aggregation The way .

Start to prepare

There are several common development tools ：

WP09hwvJUa.png

My development tools Include goland kubebuilder kustomize,kubernetes1.23.6, The work environment rocky linux 8.5 go 1.17

Be careful ： Be sure to have a look go edition The version corresponding to the development tool , As well as kubernetes Version of

kubebuilder kustomize install

https://github.com/kubernetes-sigs/kubebuilder/releases

image.png

[[email protected] ~]# wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.5.0/kubebuilder_linux_amd64

Copy

image.png

[[email protected] ~]# mv kubebuilder_linux_amd64 /usr/bin/kubebuilder
[[email protected] ~]# chmod +x /usr/bin/kubebuilder
[[email protected] ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.5.0", KubernetesVendor:"1.24.1", GitCommit:"26d12ab1134964dbbc3f68877ebe9cf6314e926a", BuildDate:"2022-06-24T12:17:52Z", GoOs:"linux", GoArch:"amd64"}

Copy

[email protected] ~]# wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.5.5/kustomize_v4.5.5_linux_amd64.tar.gz

Copy

image.png

[[email protected] ~]# tar zxvf kustomize_v4.5.5_linux_amd64.tar.gz 
kustomize
[[email protected] ~]# chmod +x kustomize
[[email protected] ~]# mv kustomize /usr/bin/kustomize
[[email protected] ~]# kustomize version
{Version:kustomize/v4.5.5 GitCommit:daa3e5e2c2d3a4b8c94021a7384bfb06734bcd26 BuildDate:2022-05-20T20:25:40Z GoOs:linux GoArch:amd64}

Copy

wITLS3q5GK.png

Create and initialize project

goland Create a kube-oprator1 Project ：

1yCLA8PmOe.png

The terminal executes a command ：

[[email protected] kube-oprator1]$ kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"

Copy

bLv3XJ03N8.png

It seems to remind me go Version is too low ？（go edition 1.17.6 My is ）

jHNIjmd96p.png

Upgrade go edition

Be careful ： Not necessary , The latter is lowered kubebuilder Version of .go Keep the version 1.17 Version of the ,

Browser open https://golang.google.cn/dl/ go The download page , choice 1.17 Download the latest version and replace the local GO edition ！

image.png

[[email protected] ~]# wget https://golang.google.cn/dl/go1.17.11.linux-amd64.tar.gz
[[email protected] ~]# tar zxvf go1.17.11.linux-amd64.tar.gz
[[email protected] ~]# which go
/usr/go/bin/go
[[email protected] ~]# cd go/
[[email protected] ~]# cp -Ra * /usr/go/
[[email protected] go]# go version go1.17.11 linux/amd64

Copy

.... It seems that the report is wrong , Took a closer look https://github.com/kubernetes-sigs/kubebuilder/releases I'd better change it kubebuilder Version ......

image.png

kubebuilder edition 3.4.1

[[email protected] ~]# wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.4.1/kubebuilder_linux_amd64
[[email protected] ~]# mv kubebuilder_linux_amd64 /usr/bin/kubebuilder
mv： Is it covered? '/usr/bin/kubebuilder'？ y
[[email protected] ~]# chmod +x /usr/bin/kubebuilder 
[[email protected] ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.4.1", KubernetesVendor:"1.23.5", GitCommit:"d59d7882ce95ce5de10238e135ddff31d8ede026", BuildDate:"2022-05-06T13:58:56Z", GoOs:"linux", GoArch:"amd64"}

Copy

kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"

Copy

image.png

The structure of the generated directory is as follows ：

image.png

Focus on config/default/ kustomization.yaml file ：

image.png

Now you can understand the configuration , Namespace ！ I will not change the default configuration here ！

 [[email protected] kube-oprator1]$ kubebuilder create api --group myapp1 --version v1 --kind Redis

Copy

ScOkSanZMO.png

The directory structure is as follows

image.png

Be careful : About domain group version kind Corresponding :

apiVersion:myapp1.zhangpeng.com/v1
kind: Redis

Copy

image.png

Simply create one crd

api/v1/redis_type.go

quaCpibxLU.png

7thoxb574k.png

Show me how to delete Foo Field , Add one Port Field , Set up port Field is int type ！

image.png

With test Under the table of contents yaml File customization crd

test/redis.yaml

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1011

Copy

make install establish crd

d9VEyF4yku.png

[[email protected] kube-oprator1]$ kubectl get crd
No resources found
[[email protected] kube-oprator1]$ make install 
GOBIN=/home/zhangpeng/GolandProjects/kube-oprator1/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
/home/zhangpeng/GolandProjects/kube-oprator1/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/zhangpeng/GolandProjects/kube-oprator1/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp1.zhangpeng.com created
[[email protected] kube-oprator1]$ kubectl get crd
NAME                         CREATED AT
redis.myapp1.zhangpeng.com   2022-06-28T06:44:52Z

Copy

About reconcile

controllers/redis_controller.go

About reconcile Just don't ask for a better understanding

KRuf7XRqfW.png

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	redis := &myapp1v1.Redis{}
	if err := r.Get(ctx, req.NamespacedName, redis); err != nil {
		fmt.Println(err)
	} else {
		fmt.Println("object", redis)
	}

	return ctrl.Result{}, nil
}

Copy

image.png

Local debugging make run

As soon as the terminal is running

[[email protected] kube-oprator1]$ maker run  

Copy

terminal 2 function

 
 [[email protected] kube-oprator1]$ kubectl apply -f test/redis.yaml

Copy

Observation terminal 1 The output is as follows ：

image.png

Preliminary release to kubernetes colony

notes ： My environment has installed podman, About podman To baidu , The mirror warehouse uses Tencent cloud image warehouse Personal Edition

About Podman

First modify docker The build command is podman!

image.png

podman login Remember the password .... Basic heel docker You can use it in the same way

[[email protected] kube-oprator1]$ podman login --username=xxxxx ccr.ccs.tencentyun.com

Copy

dockerhub Speed up

In particular dockerhub Speed up

[[email protected] kube-oprator1]$  vim /etc/containers/registries.conf

Copy

The acceleration address is added at the end of the file ！

short-name-mode = "permissive"
[[registry]]
prefix="docker.io"
location="pvurwzu6.mirror.aliyuncs.com"

Copy

restart podman service

[[email protected] kube-oprator1]$  systemctl restart podman

Copy

Build publishing image

Dockerfile Add... To the file GOPROXY

ENV GOPROXY=https://goproxy.io

Copy

[[email protected] kube-oprator1]$ make docker-build docker-push IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

Copy

zdGHw4xpxA.png

Incidentally, I found that one of Tencent cloud's personal warehouses is not displayed OCI-Image The size of bug.....

RBXC4UY1Vv.png

meokWhyqz0.png

notes ： The process is tortuous . In the middle, there is science that doesn't move under the mirror on the Internet , Autonomic brain . Such as “gcr.io/distroless/static:nonroot Mirror my operating environment as rocky linux 8.5 When I couldn't download it, I went online directly ......

How to publish ：

[[email protected] kube-oprator1]$ make deploy IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

Copy

make Failed again, finally according to Makefile in deploy Manually execute the following commands ：

[[email protected] kube-oprator1]$ cd config/manager &&  kustomize edit set image controller=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ kustomize build config/default | kubectl apply -f -

Copy

Be careful ： Both commands are in kube-oprator1 Executed under the project root directory ！

image.png

image.png

The default namespace is not modified kube-oprator1-system namespace Under the namespace pod state ！

[[email protected] kube-oprator1]$ kubectl get ns
NAME                   STATUS   AGE
default                Active   61d
kube-node-lease        Active   61d
kube-oprator1-system   Active   25h
kube-public            Active   61d
kube-system            Active   61d
zhangpeng1             Active   8d
[[email protected] kube-oprator1]$ kubectl get pods -n kube-oprator1-system

Copy

Theoretically pod The deployment was not successful , Here's why ：gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0 Unable to download ...... I use benzene method ,rocky Development machine science downloads images online and uploads them to Tencent and image warehouse , then pull Mirror to kuberntes machine work node . Yes, of course work Node i only have one test environment, which is OK ～

image.png

wait for pod running

2022-06-28 18-37-36 Screen capture of .png

CRD Custom resource simple validation

With test/redis.yaml For example

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1011

Copy

reference https://book.kubebuilder.io/reference/markers/crd-validation.html Just set it up port The scope of the ！

image.png

	// +kubebuilder:validation:Minimum:=1024
	// +kubebuilder:validation:Maximum:=10240

Copy

image.png

make install Still failed ！ Or manual command

notes ： I made mistakes in this place several times , The reason is that I thought make install =kustomize build config/crd | kubectl apply -f -, Took a closer look Makefile：

image.png

contain manifests Steps for , I tried it, and it worked ！

[[email protected] kube-oprator1]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[[email protected] kube-oprator1]$ kustomize build config/crd | kubectl apply -f -

Copy

image.png

[[email protected] kube-oprator1]$ kubectl get crd redis.myapp1.zhangpeng.com -o yaml

Copy

2N5nbJSMqC.png

[[email protected] kube-oprator1]$ kubectl delete -f test/redis.yaml
[[email protected] kube-oprator1]$ kubectl apply -f test/redis.yaml

Copy

The output is as follows ： The port is less than 1024 Cannot create successfully

6bpr7hgcdQ.png

modify test/redis.yaml port: 1024

apiVersion: myapp1.zhangpeng.com/v1
kind: Redis
metadata:
  name: myapp
spec:
  port: 1024

Copy

wrUgeuqwvC.png

[[email protected] cert]$ kubectl get pods -A|grep cert
cert-manager           cert-manager-677874db78-zcm6l                       1/1     Running            0                  14m
cert-manager           cert-manager-cainjector-6c5bf7b759-mf4gf            1/1     Running            0                  14m
cert-manager           cert-manager-webhook-5685fdbc4b-ncrxl               1/1     Running            0                  14m

Copy

webhook A simple test

Simple access controller webhook create

[[email protected] kube-oprator1]$ kubebuilder create webhook --group myapp1 --version v1 --kind Redis --defaulting --programmatic-validation

Copy

bVzxTxybrI.png

kube-oprator1 api/v1 Added... To the directory webhook Related documents of , Made a simple verification name=zhangpeng

image.png

func (r *Redis) ValidateCreate() error {
	redislog.Info("validate create", "name", r.Name)
	if r.Name == "zhangpeng" {
		return errors.New("error name")
	}

	// TODO(user): fill in your validation logic upon object creation.
	return nil
}

Copy

Certificate management cert-manager:

visit https://github.com/cert-manager/cert-manager/releases The download page ,1.19.0 yes alpha edition I used it 1.18.2 Version of ！

image.png

[[email protected] cert]$ pwd
/home/zhangpeng/cert
[[email protected] cert]$ wget https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
[[email protected] cert]$ kubectl apply -f cert-manager.yaml
[[email protected] cert]$ kubectl get pods -A|grep cert

Copy

2022-06-28 19-50-45 Screen capture of .png

2022-06-28 19-53-21 Screen capture of .png

Be careful ： Mirror images still need to be downloaded scientifically

Modify file

config/default/kustomization.yaml The down arrow callout section unlocks the annotation

image.png

image.png

config/manager/manager.yaml

k4tZYzr46h.png

Delete crd

make uninstall Yes, but mine make Always fail ...... Delete... Directly ！

[[email protected] kube-oprator1]$kubectl delete crd redis.myapp1.zhangpeng.com

Copy

ekWrnYeRSP.png

Packaging image publishing

Packaging and publishing images , In fact, it is better to modify an image label tag, Here is a demonstration That's it first ！make install I don't know there is a problem Direct copy Makefile The order in ！ Build an image and publish it ！

[[email protected] kube-oprator1]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[[email protected] kube-oprator1]$ kustomize build config/crd  | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp1.zhangpeng.com configured

[[email protected] kube-oprator1]$ make docker-build docker-push IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ make deploy IMG=ccr.ccs.tencentyun.com/layatools/zpredis:v1

Copy

2022-06-28 20-12-47 Screen capture of .png

djU8z66Qcf.png

[[email protected] kube-oprator1]$ cd config/manager && kustomize edit set image controller=ccr.ccs.tencentyun.com/layatools/zpredis:v1
[[email protected] kube-oprator1]$ kustomize build config/default | kubectl apply -f -

Copy

image.png

image.png

EN, modified to zhangpeng1

image.png

Created successfully ？ to glance at make run,but make run Can't run ？

image.png

The local debug mode is commented out main.go SetupWebhookWithManager

image.png

image.png

image.png

summary ：

1. Pay attention to the version matching between development tools

2.make Look at it when you fail Makefile You can manually run the relevant commands in

3. Resource cleanup , Local debugging mode

4. Next, we are going to design a simple oprator?