当前位置：网站首页>Kubernetes certificate collection

Kubernetes certificate collection

2022-06-11 13:17:00 【Look, future】

List of articles
PKI certificate
How many certificates are there in total ？
Why the same “ set ” The certificate in must be the same CA Signed ？
TLS bootstrapping
kubelet Why certificates are different
Formal production certificate
Certificates to be prepared
CFSSL
establish CA certificate
Create a directory for storing certificates
Create a Certificate profile
establish CA Certificate signature request file
Generate CA Certificate and private key
establish kubernetes certificate
establish kubernetes Certificate signature request file kubernetes-csr.json
Generate kubernetes Certificate and private key
establish admin Certificate signature request file admin-csr.json
Generate admin Certificate and private key
establish kube-proxy certificate
establish kube-proxy Certificate signature request file kube-proxy-csr.json
Generate kube-proxy Certificate and private key
View certificate information
k8s file

PKI certificate

Kubernetes need PKI Certificate to pass TLS Authentication . If you use kubeadm install Kubernetes, The certificate required by the cluster will be automatically generated . You can also generate your own certificates , for example , By not storing the private key in API It is safer to keep the private key on the server . Of course , We are currently installing it manually .

There is something wrong with the installation , Say the certificate is illegal , So I thought , Let's sort out the certificates first , Otherwise, such problems may arise later , It's not easy to check .

How many certificates are there in total ？

First from Etcd Count up ：

1、Etcd External services , Have a set of etcd server certificate 

2、Etcd Communication between nodes , Have a set of etcd peer certificate 

3、Kube-APIserver visit Etcd, Have a set of etcd client certificate

Count again kubernetes：

4、Kube-APIserver External services , Have a set of kube-apiserver server certificate 

5、kube-scheduler、kube-controller-manager、kube-proxy、kubelet And other components that may be used ,
   Need to access kube-APIserver, Have a set of kube-APIserver client certificate 

6、kube-controller-manager To generate a service service account,
   There must be a pair to sign service account Certificate (CA certificate )

7、kubelet External services , Have a set of kubelet server certificate 

8、kube-APIserver Need to access kubelet, Have a set of kubelet client certificate

All in all 8 set .

Certificates in the same set must use the same CA Signed , Signing certificates in different sets CA Can be the same , It can be different . for example , all etcd server The certificate needs to be the same CA Signed , be-all etcd peer The certificate also needs to be the same CA Signed , And one etcd server Certificate and a etcd peer certificate , It could be two CA Signed by the agency , They have nothing to do with each other . These are two sets of certificates .

Why the same “ set ” The certificate in must be the same CA Signed ？

The reason lies in the verification of these certificates . Because on the side where you want to verify these certificates , Usually only one can be specified Root CA. thus , The verified certificate naturally needs to be verified by the same Root CA Corresponding private key signature , Otherwise, it cannot pass the certification .

Actually , Use a set of certificates （ All use one set CA To sign ） Can also build K8S, The same can be produced on , But sort out the relationship between these certificates , In case of certificate error , When the request is rejected , It's not impossible to start , And if the relationship between certificates is not clear , When maintaining or solving problems , Changed the certificate rashly , If you don't do it well, it will paralyze the whole system .

TLS bootstrapping

Kubernetes1.4 Version introduces a set of API. This group of API The introduction of , So that we don't have to prepare in advance kubelet Certificates used .

Every kubelet The certificates used are unique , Because it needs to bind their respective IP Address , So you need to give everyone kubelet Make the certificate separately , If the business volume is large ,node There will be many nodes , thus kubelet The number of has also increased , And it will change frequently （ Increase or decrease Node）kubelet Certificate making has become a very troublesome thing . Use TLS bootstrapping You can save a lot of trouble .

working principle ：Kubelet At first start-up , First use the same bootstrap token As proof . This token Has been set in advance to belong to the user group system：bootstrappers, And the permissions of this user group are limited to applying for certificates . Use this bootstrap token After certification ,kubelet Apply for their own two sets of certificates （kubelet server、kube-apiserver client for kubelet）, After successful application , Then use your own certificate for authentication , So I have kubelet Due authority . thus , It removes the manual for each kubelet The process of preparing certificates , also kubelet The certificate of can also be updated automatically

Official document reference ：https://kubernetes.io/docs/tasks/tls/certificate-rotation/

kubelet Why certificates are different

This is done for the purpose of auditing , The other is for safety . Every kubelet It's a server （kube-apiserver Need to access kubelet）, It's also a client （kubelet Need to access kube-apiserver）, Therefore, there should be two sets of certificates for the server and the client .

The server certificate needs to be bound to the server address , Every kubelet The addresses of are different , Even binding a domain name is binding a different domain name , Therefore, the server addresses are different

The client certificate should not be the same , Every kubelet And the certificate of the machine IP After binding , Can prevent a kubelet After the disclosure of the certificate of certification , Pass a request forged from another machine .

security , If each node Reserved for signing certificates on bootstrap token, that bootstrap token After the leak , Can you sign the certificate at will ？ The security risks are very big . therefore ,kubelet After the startup is successful , The local bootstrap token Need to be deleted .

Formal production certificate

Although you can use multiple sets of certificates , But maintain multiple sets CA It is too complicated , Here's still a CA Sign all certificates .

Certificates to be prepared

admin-key.pem

admin.pem

ca-key.pem

ca.pem

kube-proxy-key.pem

kube-proxy.pem

kubernetes-key.pem

kubernetes.pem

The components that use certificates are as follows ：

etcd： Use  ca.pem、kubernetes-key.pem、kubernetes.pem

kube-apiserver： Use  ca.pem、kubernetes-key.pem、kubernetes.pem

kubelet： Use  ca.pem

kube-proxy： Use  ca.pem、kube-proxy-key.pem、kube-proxy.pem

kubectl： Use  ca.pem、admin-key.pem、admin.pem

kube-controller-manager： Use  ca-key.pem、ca.pem

There's this one , There is also a direction for troubleshooting when there is a problem with the certificate .

CFSSL

We use CFSSL To make a certificate , It is cloudflare Development of an open source PKI Tools , It is a complete system CA Service system , May sign 、 Revocation of certificates, etc , It covers the entire life cycle of a certificate , Only its command-line tools are used later .

notes ： In general ,K8S The certificate in only needs to be created once , In the future, when adding new nodes to the cluster, just set /etc/kubernetes/ssl Copy the certificate under the directory to the new node .

Download and install cfssl Command line tools

# The following operation is only in master on 
# Go to the download directory 
cd /opt/TLS/download
# Download and unzip cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
chmod +x cfssl*
[[email protected] download]# ll
total 40232
-rwxr-xr-x 1 root root 16659824 Dec  7 15:36 cfssl_1.6.1_linux_amd64
-rwxr-xr-x 1 root root 13502544 Dec  7 15:35 cfssl-certinfo_1.6.1_linux_amd64
-rwxr-xr-x 1 root root 11029744 Dec  7 15:35 cfssljson_1.6.1_linux_amd64

# Only in master On the operation 
cd /opt/TLS/download
cp cfssl_1.6.1_linux_amd64 /usr/local/bin/cfssl
cp cfssljson_1.6.1_linux_amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_1.6.1_linux_amd64 /usr/local/bin/cfssl-certinfo
[[email protected] download]# ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 16659824 Apr  4 08:46 /usr/local/bin/cfssl
-rwxr-xr-x 1 root root 13502544 Apr  4 08:46 /usr/local/bin/cfssl-certinfo
-rwxr-xr-x 1 root root 11029744 Apr  4 08:46 /usr/local/bin/cfssljson

establish CA certificate

Create a directory for storing certificates

[[email protected] bin]# mkdir -p /opt/kubernetes/ssl/
[[email protected] bin]# cd /opt/kubernetes/ssl/

Create a Certificate profile

[[email protected] ssl]# vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

Field description ：

ca-config.json： Multiple can be defined  profiles, Specify different expiration times 、 Use the scene and other parameters ; Use some of the following when signing the certificate  profile;

signing： Indicates that this certificate can sign other certificates ; Generated ca.pem In the certificate  CA=TRUE;

server auth： Express client You can use this.  CA  Yes server Certificate provided to verify ;

client auth： Express server You can use this. CA Yes client Certificate provided to verify ;

expiry： Expiration time

establish CA Certificate signature request file

[[email protected] ssl]# vim ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ],
    "ca": {
       "expiry": "87600h"
    }
}

Field description ：

“CN”：Common Name,kube-apiserver  Extract this field from the certificate as the requested user name  (User Name); The browser uses this field to verify whether the website is legal ;

“O”：Organization,kube-apiserver  Extract this field from the certificate as the group to which the requesting user belongs  (Group);

Generate CA Certificate and private key

[[email protected]8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[[email protected] ssl]# ls | grep ca
ca-config.json
ca.csr
ca-csr.json
ca-key.pem
ca.pem

among ca-key.pem yes ca The private key ,ca.csr Is a signing request ,ca.pem yes CA certificate , It's the back kubernetes Components will be used RootCA.

establish kubernetes certificate

establish kubernetes Certificate signature request file kubernetes-csr.json

[[email protected] ssl]# vim kubernetes-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.214.88",
      "192.168.214.89",
      "192.168.214.90",
      "192.168.214.200",
      "192.168.214.201",
      "192.168.214.202",
      "10.254.0.1",
      "192.168.214.210",
      "192.168.214.1/24",
      "kubernetes",
      "kube-api.wangk8s-master.com",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

If hosts If the field is not empty, you need to specify the IP Or domain name list .

Since the certificate was subsequently etcd Clusters and kubernetes master Use , take etcd、master Node IP All fill in , As well as service Network first IP.( It's usually kube-apiserver designated service-cluster-ip-range The first network segment IP, Such as 10.254.0.1)

My settings here include a private image repository , Three etcd, Three master, Of the above physical nodes IP It can also be replaced with the host name .

Generate kubernetes Certificate and private key

[[email protected] ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
[[email protected] ssl]# ls |grep kubernetes
kubernetes.csr
kubernetes-csr.json
kubernetes-key.pem
kubernetes.pem

establish admin Certificate signature request file admin-csr.json

[[email protected] ssl]# admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

explain ：

follow-up kube-apiserver Use RBAC Client side ( Such as kubelet、kube-proxy、Pod) Request authorization ;

kube-apiserver Some of them are predefined RBAC The use of RoleBindings, Such as cluster-admin take Group system:masters And Role cluster-admin binding , The Role Call granted kube-apiserver All of the API Authority ;

O Specify the Group by system:masters,kubelet Use this certificate to access kube-apiserver when , Because of the certificate CA Signature , So the certification passed , At the same time, the certificate user group is pre authorized system:masters, So was granted access to all API Authority ;

notes ： This admin certificate , It will be used by the administrator in the future kube config Configuration file , Now we generally recommend using RBAC Come on kubernetes Control role permissions , kubernetes Put... In the certificate CN Field As User, O Field as Group

Generate admin Certificate and private key

[[email protected] ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
[[email protected] ssl]# ls | grep admin
admin.csr
admin-csr.json
admin-key.pem
admin.pem

establish kube-proxy certificate

establish kube-proxy Certificate signature request file kube-proxy-csr.json

[[email protected] ssl]# vim kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

CN Specify the User by system:kube-proxy;

kube-apiserver Predefined RoleBinding system:node-proxier take User system:kube-proxy And Role system:node-proxier binding , The Role Call granted kube-apiserver Proxy relevant API Authority ;

Generate kube-proxy Certificate and private key

[[email protected] ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
[[email protected] ssl]# ls |grep kube-proxy
kube-proxy.csr
kube-proxy-csr.json
kube-proxy-key.pem
kube-proxy.pem

After the above operations , We will use the following files ：

[[email protected] ssl]# ls | grep pem
admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-proxy-key.pem
kube-proxy.pem
kubernetes-key.pem
kubernetes.pem

View certificate information

[[email protected] ssl]# cfssl-certinfo -cert kubernetes.pem
{
  "subject": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "issuer": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "serial_number": "321233745860282370502438768971300435157761820875",
  "sans": [
    "192.168.214.1/24",
    "kubernetes",
    "kube-api.wangk8s-master.com",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "127.0.0.1",
    "192.168.214.88",
    "192.168.214.89",
    "192.168.214.90",
    "192.168.214.200",
    "192.168.214.201",
    "192.168.214.202",
    "10.254.0.1",
    "192.168.214.210"
  ],
  "not_before": "2019-03-12T11:26:00Z",
  "not_after": "2029-03-09T11:26:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "CB:34:54:33:1F:F4:37:E:E5:94:B7:F5:8A:3D:F4:A4:43:43:E2:7F",
  "subject_key_id": "EC:31:D8:5F:4:E3:6F:C2:7F:DA:A8:F0:BD:A:B9:1F:56:7B:9A:DF",
  "pem": "-----BEGIN CERTIFICATE-----\nM（ Omit here ）=\n-----END CERTIFICATE-----\n"
}

In the building k8s In clusters , Distribute these files to other node machines in this cluster . thus ,TLS Certificate creation completed .