Develop a controller that prohibits deleting namespaces

WeChat official account ： O & M development story , author ： Jock

Hello everyone , I'm Jock .

image.png

I received a message from a friend yesterday , Say you accidentally put the cluster business namespace It's gone , As a result, the whole business has stagnated , Ask me if it is forbidden to delete namespace The plan .

In my memory ,Kubernetes There is no such controller in the access of , So I told him that I need to develop an access controller to achieve my goal .

As a person , What is right ！ I can't just take off my pants , Don't fart . So here is how to customize Kubernetes Access controller for .

The theory is introduced

Access controller （Admission Controller） be located API Server in , Before the object is persisted , The admission controller intercepts the pair API Server Request , Generally used for authentication and authorization . It contains two special controllers ：MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

• MutatingAdmissionWebhook ： Used to change the request object , such as istio For each Pod Inject sidecar, It is through it that .

• ValidatingAdmissionWebhook： Used to verify the request object

The flow of the whole access controller is as follows ：

When API When requesting access ,mutating and validating The controller uses an external in the configuration webhooks List concurrent calls , The following rules ：

• If all webhooks Approve request , Access control chain continues to flow .

• If there is any one webhooks Stop the request , Then the admission control request is terminated , And return to the first webhook Reason for blocking . among , Multiple webhooks Blocking will only return the first webhook Reason for blocking .

• If you're calling webhook There was a mistake in the process , Then the request will be terminated or ignored webhook.

The access controller is in API Server Configured in the startup parameters of . An admission controller may belong to one of the above two , Or maybe both belong to .

We are deploying Kubernetes When clustering, a series of admission controllers will be enabled by default , If these access controllers are not set, you can say Kubernetes The cluster is running naked , You should ask the administrator to add an admission controller to the cluster .

Code implementation

Implementation logic

Before development, we should have a general understanding of the access controller Webhook The general implementation logic of ：

• Webhook It's a standard HTTP service , receive HTTP request

• The received request is a AdmissionReview object

• Then we customize Hook I'll deal with this AdmissionReview object

• After processing, return to another AdmissionReview object , This will include the processing results

AdmissionReview The structure of is as follows ：

// AdmissionReview describes an admission review request/response.
type AdmissionReview struct {
 metav1.TypeMeta `json:",inline"`
 // Request describes the attributes for the admission request.
 // +optional
 Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`
 // Response describes the attributes for the admission response.
 // +optional
 Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`
}

It is clear from the naming of the code , Before the request is sent to WebHook We only need to pay attention to the internal AdmissionRequest( Actual input ), In what we wrote WebHook After processing, you only need to return the containing AdmissionResponse( Actual return body ) Of AdmissionReview Object can ; in general AdmissionReview The object is a shell , The request is inside AdmissionRequest, The response is inside AdmissionResponse.

Concrete realization

（1） First create a HTTP Server, Listening port , Receiving request

package main

import (
    "context"
    "flag"
    "github.com/joker-bai/validate-namespace/http"
    log "k8s.io/klog/v2"
    "os"
    "os/signal"
    "syscall"
)

var (
    tlscert, tlskey, port string
)

func main() {
    flag.StringVar(&tlscert, "tlscert", "/etc/certs/cert.pem", "Path to the TLS certificate")
    flag.StringVar(&tlskey, "tlskey", "/etc/certs/key.pem", "Path to the TLS key")
    flag.StringVar(&port, "port", "8443", "The port to listen")
    flag.Parse()
    
    server := http.NewServer(port)
    go func() {
        if err := server.ListenAndServeTLS(tlscert, tlskey); err != nil {
            log.Errorf("Failed to listen and serve: %v", err)
        }
    }()
    
    log.Infof("Server running in port: %s", port)
    
    // listen shutdown signal
    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
    <-signalChan
    
    log.Info("Shutdown gracefully...")
    if err := server.Shutdown(context.Background()); err != nil {
        log.Error(err)
    }
}

Due to access controller and Webhook You need to use TLS communicate , So the port monitored above is TLS port , adopt server.ListenAndServeTLS Realization , Later, when deploying services, you need to hang the certificate in the corresponding directory .

（2） Definition Handler, Distribute requests to specific processing methods

package http

import (
 "fmt"
 "github.com/joker-bai/validate-namespace/namespace"
 "net/http"
)

// NewServer creates and return a http.Server
func NewServer(port string) *http.Server {
 // Instances hooks
 nsValidation := namespace.NewValidationHook()

 // Routers
 ah := newAdmissionHandler()
 mux := http.NewServeMux()
 mux.Handle("/healthz", healthz())
 mux.Handle("/validate/delete-namespace", ah.Serve(nsValidation))

 return &http.Server{
  Addr:    fmt.Sprintf(":%s", port),
  Handler: mux,
 }
}

Realization admissionHandler, The main function is to http body The content of is parsed into AdmissionReview object , Then call the specific Hook Handle , Then put the result in AdmissionReview in , Return to the client .

package http

import (
 "encoding/json"
 "fmt"
 "io"
 "net/http"

 "github.com/douglasmakey/admissioncontroller"

 "k8s.io/api/admission/v1beta1"
 admission "k8s.io/api/admission/v1beta1"
 meta "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/apimachinery/pkg/runtime"
 "k8s.io/apimachinery/pkg/runtime/serializer"
 log "k8s.io/klog/v2"
)

// admissionHandler represents the HTTP handler for an admission webhook
type admissionHandler struct {
 decoder runtime.Decoder
}

// newAdmissionHandler returns an instance of AdmissionHandler
func newAdmissionHandler() *admissionHandler {
 return &admissionHandler{
  decoder: serializer.NewCodecFactory(runtime.NewScheme()).UniversalDeserializer(),
 }
}

// Serve returns a http.HandlerFunc for an admission webhook
func (h *admissionHandler) Serve(hook admissioncontroller.Hook) http.HandlerFunc {
 return func(w http.ResponseWriter, r *http.Request) {
  w.Header().Set("Content-Type", "application/json")
  if r.Method != http.MethodPost {
   http.Error(w, fmt.Sprint("invalid method only POST requests are allowed"), http.StatusMethodNotAllowed)
   return
  }

  if contentType := r.Header.Get("Content-Type"); contentType != "application/json" {
   http.Error(w, fmt.Sprint("only content type 'application/json' is supported"), http.StatusBadRequest)
   return
  }

  body, err := io.ReadAll(r.Body)
  if err != nil {
   http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)
   return
  }

  var review admission.AdmissionReview
  if _, _, err := h.decoder.Decode(body, nil, &review); err != nil {
   http.Error(w, fmt.Sprintf("could not deserialize request: %v", err), http.StatusBadRequest)
   return
  }

  if review.Request == nil {
   http.Error(w, "malformed admission review: request is nil", http.StatusBadRequest)
   return
  }

  result, err := hook.Execute(review.Request)
  if err != nil {
   log.Error(err)
   w.WriteHeader(http.StatusInternalServerError)
   return
  }

  admissionResponse := v1beta1.AdmissionReview{
   Response: &v1beta1.AdmissionResponse{
    UID:     review.Request.UID,
    Allowed: result.Allowed,
    Result:  &meta.Status{Message: result.Msg},
   },
  }

  res, err := json.Marshal(admissionResponse)
  if err != nil {
   log.Error(err)
   http.Error(w, fmt.Sprintf("could not marshal response: %v", err), http.StatusInternalServerError)
   return
  }

  log.Infof("Webhook [%s - %s] - Allowed: %t", r.URL.Path, review.Request.Operation, result.Allowed)
  w.WriteHeader(http.StatusOK)
  w.Write(res)
 }
}

func healthz() http.HandlerFunc {
 return func(w http.ResponseWriter, r *http.Request) {
  w.WriteHeader(http.StatusOK)
  w.Write([]byte("ok"))
 }
}

The above processing is through hook.Execute To process requests , This is a admissionController A structure implemented internally , It defines a method for each operation , as follows ：

// AdmitFunc defines how to process an admission request
type AdmitFunc func(request *admission.AdmissionRequest) (*Result, error)

// Hook represents the set of functions for each operation in an admission webhook.
type Hook struct {
 Create  AdmitFunc
 Delete  AdmitFunc
 Update  AdmitFunc
 Connect AdmitFunc
}

We need to achieve specific AdmitFunc, And register .

（3） Register the method implemented by yourself to Hook in .

package namespace

import (
 "github.com/douglasmakey/admissioncontroller"
)

// NewValidationHook delete namespace validation hook
func NewValidationHook() admissioncontroller.Hook {
 return admissioncontroller.Hook{
  Delete: validateDelete(),
 }
}

（4） To achieve concrete AdmitFunc

package namespace

import (
 "github.com/douglasmakey/admissioncontroller"
 log "k8s.io/klog/v2"

 "k8s.io/api/admission/v1beta1"
)

func validateDelete() admissioncontroller.AdmitFunc {
 return func(r *v1beta1.AdmissionRequest) (*admissioncontroller.Result, error) {
  if r.Kind.Kind == "Namespace" {
   log.Info("You cannot delete namespace: ", r.Name)
   return &admissioncontroller.Result{Allowed: false}, nil
  } else {
   return &admissioncontroller.Result{Allowed: true}, nil
  }
 }
}

The implementation here is very simple , If Kind by Namespace, Reject the operation .

The deployment of test

Business logic development has been completed above , Next, deploy it to Kubernetes Test the cluster .

Deploy

（1） To write Dockerfile, Package the application as a mirror

FROM golang:1.17.5 AS build-env
ENV GOPROXY https://goproxy.cn
ADD . /go/src/app
WORKDIR /go/src/app
RUN go mod tidy
RUN cd cmd && GOOS=linux GOARCH=amd64 go build -v -a -ldflags '-extldflags "-static"' -o /go/src/app/app-server /go/src/app/cmd/main.go

FROM registry.cn-hangzhou.aliyuncs.com/coolops/ubuntu:22.04
ENV TZ=Asia/Shanghai
COPY --from=build-env /go/src/app/app-server /opt/app-server
WORKDIR /opt
EXPOSE 80
CMD [ "./app-server" ]

（2） establish TLS certificate , Use scripts to create

#!/bin/bash

set -e

usage() {
    cat <<EOF
Generate certificate suitable for use with an sidecar-injector webhook service.

This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with sidecar-injector webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.

The server key/cert k8s CA cert are stored in a k8s secret.

usage: ${0} [OPTIONS]

The following flags are required.

       --service          Service name of webhook.
       --namespace        Namespace where webhook service and secret reside.
       --secret           Secret name for CA certificate and server certificate/key pair.
EOF
    exit 1
}

while [[ $# -gt 0 ]]; do
    case ${1} in
        --service)
            service="$2"
            shift
            ;;
        --secret)
            secret="$2"
            shift
            ;;
        --namespace)
            namespace="$2"
            shift
            ;;
        *)
            usage
            ;;
    esac
    shift
done

[ -z ${service} ] && service=validate-delete-namespace
[ -z ${secret} ] && secret=validate-delete-namespace-tls
[ -z ${namespace} ] && namespace=default

if [ ! -x "$(command -v openssl)" ]; then
    echo "openssl not found"
    exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf

# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true

# create  server cert/key CSR and  send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# verify CSR has been created
while true; do
    kubectl get csr ${csrName}
    if [ "$?" -eq 0 ]; then
        break
    fi
done

# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=${tmpdir}/server-key.pem \
        --from-file=cert.pem=${tmpdir}/server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

（3） To write Deployment Deployment Services

apiVersion: apps/v1
kind: Deployment
metadata:
  name: validate-delete-namespace
  labels:
    app: validate-delete-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: validate-delete-namespace
  template:
    metadata:
      labels:
        app: validate-delete-namespace
    spec:
      containers:
        - name: server
          image: registry.cn-hangzhou.aliyuncs.com/coolops/validate-delete-namespace:latest
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8443
              scheme: HTTPS
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: tls-certs
              mountPath: /etc/certs
              readOnly: true
      volumes:
        - name: tls-certs
          secret:
            secretName: validate-delete-namespace-tls
---
apiVersion: v1
kind: Service
metadata:
  name: validate-delete-namespace
spec:
  selector:
    app: validate-delete-namespace
  ports:
    - port: 443
      targetPort: 8443

（4） Deploy Webhook

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validate-delete-namespace
webhooks:
  - name: validate-delete-namespace.default.svc.cluster.local
    clientConfig:
      service:
        namespace: default
        name: validate-delete-namespace
        path: "/validate/delete-namespace"
      caBundle: "${CA_BUNDLE}"
    rules:
      - operations:
          - DELETE
        apiGroups:
          - ""
        apiVersions:
          - "v1"
        resources:
          - namespaces
    failurePolicy: Ignore

Here's one ${CA_BUNDLE} Place holder , Creating Webhook Replace it when , Use the following command ：

cat ./validate-delete-namespace.yaml | sh ./patch-webhook-ca.sh > ./webhook.yaml

Then create webhook.yaml that will do .

kubectl apply -f webhook.yaml

All the above files are in the code base , You can deploy directly using scripts .

# sh deploy.sh 
creating certs in tmpdir /tmp/tmp.SvMHWcPI6x 
Generating RSA private key, 2048 bit long modulus
..........................................+++
.............................................................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default created
NAME                                AGE   REQUESTOR          CONDITION
validate-delete-namespace.default   0s    kubernetes-admin   Pending
certificatesigningrequest.certificates.k8s.io/validate-delete-namespace.default approved
secret/validate-delete-namespace-tls created
Creating k8s admission deployment
deployment.apps/validate-delete-namespace created
service/validate-delete-namespace created
validatingwebhookconfiguration.admissionregistration.k8s.io/validate-delete-namespace created

After execution , You can view specific information .

# kubectl get po
NAME                                         READY   STATUS    RESTARTS   AGE
validate-delete-namespace-74c9b8b7bd-5g9zv   1/1     Running   0          3s
# kubectl get secret
NAME                            TYPE                                  DATA   AGE
default-token-kx5wf             kubernetes.io/service-account-token   3      72d
validate-delete-namespace-tls   Opaque                                2      53s
# kubectl get ValidatingWebhookConfiguration
NAME                                  CREATED AT
validate-delete-namespace             2022-06-24T09:39:26Z

test

（1） Start by opening webhook Of pod journal

# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f
I0624 17:39:27.858753       1 main.go:30] Server running in port: 8443

（2） Create a namespace And delete

# kubectl create ns joker
# kubectl get ns | grep joker
joker                             Active   4h5m
# kubectl delete ns joker
Error from server: admission webhook "validate-delete-namespace.default.svc.cluster.local" denied the request without explanation
# kubectl get ns | grep joker
joker                             Active   4h5m

It can be found that our deletion operation was rejected , And look at namespace There is still .

We can also check in the log , as follows ：

# kubectl logs validate-delete-namespace-74c9b8b7bd-5g9zv -f
I0624 17:39:27.858753       1 main.go:30] Server running in port: 8443
2022/06/24 17:43:34 You cannot delete namespace:  joker
I0624 17:43:34.664945       1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false
2022/06/24 17:43:34 You cannot delete namespace:  joker
I0624 17:43:34.667043       1 handler.go:94] Webhook [/validate/delete-namespace - DELETE] - Allowed: false

The above is a simple implementation of an access controller ,

As long as the mind does not slip , There are more ways than difficulties .

Thanks to omnipotent Baidu , Thank you for your awesome netizens .

Reference resources

• https://www.qikqiak.com/post/k8s-admission-webhook

• https://github.com/douglasmakey/admissioncontroller

• https://mritd.com/2020/08/19/write-a-dynamic-admission-control-webhook/

I am a Jock ,《 O & M development story 》 A member of the official account team. , Front line operation and maintenance workers , Cloud native practitioners , It's not just hard core technology , And our thinking and perception of Technology , Welcome to our official account , Looking forward to growing up with you ！