Read how to deploy highly available k3s with external database

Introduce

Are you deploying in high availability mode Kubernetes And encountered difficulties in configuring the remote backup database ？ This article will show you how to have an external database Postgres Of HA Deployed in configuration K3s.

K3s（https://k3s.io/） It is used for authentication of Internet of things and edge computing Kubernetes Distribution version . I deployed it in IBM Z On the virtual machine in the mainframe . I have no choice etcd, It USES Postgres As K3s Storage scheme of cluster . I blame HA The pattern is deployed Postgres. Want to know how to HA Deploy in mode Postgres Information about , Please refer to the official documentation ：https://www.postgresql.org/docs/13/high-availability.html

Postgres and K3s The communication between clusters is SSL Safe . I am here K3s server The first one 4 Layer used Nginx Load balancing . We may need to add new server nodes or shut down the server in the future . therefore , If not used K3s Server IP Instead, use a load balancer , We can avoid this kind of trouble , It can also be realized HA. Please note that ,Nginx The load balancer may also become a single point of failure . About in HA Deploy in mode Nginx, See documentation ：https://www.nginx.com/products/nginx/high-availability/

HA To configure

The following figure describes me in HA The lower part of the mode deploys K3s The way of clustering . Even one Server Something goes wrong , another Server Still accessible , And the load balancer can route all requests to accessible Server. because K3s All components of are stateless , Even if Server Something goes wrong , We will not lose any information . All information is stored in Postgres in , And must be HA.

With load balancer and database K3s HA Pattern

Environmental Science

Before we start , You need the following environment ：

operating system - SLE15SP2 virtual machine - 5 VCPU - 2 RAM - 8 GB disk - 30 GB Arch - s390x

Here are the of each virtual machine I use IP Address , You can check these at any time in this article IP The meaning of address ：

K3s Server 1 - 10.161.129.54K3s Server 2 - 10.161.129.154K3s Agent - 10.161.129.196Postgres - 10.161.129.212Nginx - 10.161.129.118

I'm on the mainframe (s390x) Deployment in virtual machine , You can also use the same commands on any other architecture . Now? , Let's take a look at the installation Postgres and K3s Cluster command .

install Postgres

I installed postgres10. We name this virtual machine Postgres. You can copy and paste the following commands to install Postgres. For safety's sake , Please make sure Postgres Database and K3s There are interactions between clusters TLS signal communication . I used a self signed certificate to identify the use of OpenSSL Created Postgres.

Postgres virtual machine

• Use zypper install Postgres package ：

zypper -n in postgresql10 postgresql10-serversystemctl start postgresql

• establish K3s database 、 User role and grant all access rights to user role ：

sudo -u postgres psqlcreate database K3s;create user K3s with encrypted password 'K3s';grant all privileges on database K3s to K3s;exit;

We will use K3s Database to store cluster information . We use K3s Users come for K3s Cluster Postgres Database authentication .

• Create logo Postgres The self signed certificates of the server and store them in /var/lib/pgsql/data/ in ：

openssl req -new -x509 -days 365 -nodes -text -out /var/lib/pgsql/data/postgres.crt   -keyout /var/lib/pgsql/data/postgres.key -subj "/CN=postgres.rancher.rke2" -addext "subjectAltName=DNS:postgres.rancher.rke2"

• Ensure that access to the private key is restricted ：

chmod 0600 /var/lib/pgsql/data/postgres.keychown postgres:postgres /var/lib/pgsql/data/postgres.key

• Mark Postgres Copy the public key certificate of to two K3s Server, send K3s Server Can verify Postgres To carry out SSL signal communication ：

scp /var/lib/pgsql/data/postgres.crt [email protected]:scp /var/lib/pgsql/data/postgres.crt [email protected]:

• take /var/lib/pgsql/data/pg_hba.conf Replace with the following ：

# TYPE  DATABASE        USER            ADDRESS                 METHOD# "local" is for Unix domain socket connections onlyflocal   all             all                                     peer# IPv4 local connections:host    all             all             127.0.0.1/32            identhostssl all             all             0.0.0.0/0               md5 clientcert=verify-full

The contents of this document indicate ,localhost No password or SSL Communication can connect to the database . however ,Postgres All other connections must be through SSL Client authentication （K3s Server yes Postgres The client of ） And password authentication to communicate .

In further setting Postgres Before the database , Let's create the logo first K3s Cluster the certificate and copy the certificate to Postgres virtual machine , In order to Postgres verification K3s Server.

K3s Server 1 virtual machine

• Create a logo K3s Cluster self signed certificate , And grant private key permission ：

openssl req -new -x509 -days 365 -nodes -text -out K3s.crt -keyout K3s.key -subj "/CN=K3s" -addext "subjectAltName=DNS:K3s"chmod 0600 K3s.key

• Copy the public key certificate to Postgres host , In order to Postgres Can verify K3s client ：

scp /home/sles/K3s.crt [email protected]:

• Copy public and private keys to other K3s Server. Two sets of Server form K3s colony ：

scp /home/sles/K3s.crt /home/sles/K3s.key [email protected]:

Next , Let's keep looking Postgres virtual machine .

Postgres virtual machine

• take K3s.crt Move to /var/lib/pgsql/data The catalogue is for Postgres Configuration file usage ：

mv /home/sles/K3s.crt /var/lib/pgsql/data/

• take /var/lib/pgsql/data/postgresql.conf The content of is modified to the following value ：

listen_addresses = '*'ssl = on#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers#ssl_prefer_server_ciphers = on#ssl_ecdh_curve = 'prime256v1'#ssl_dh_params_file = ''ssl_cert_file = '/var/lib/pgsql/data/postgres.crt'ssl_key_file = '/var/lib/pgsql/data/postgres.key'ssl_ca_file = '/var/lib/pgsql/data/K3s.crt'

listen_addresses： Set to * or Postgres Server's IP. This ensures that Postgres The server can listen to nodes IP Address .

ssl： open SSL To communicate only in a secure way .

ssl_cert_filessl_key_file： identification Postgres Certificate of database . I have created the certificate at the beginning of this article , Now just point them to the certificate location .

ssl_ca_file： This is a CA（ Certification authority ） certificate , Used to identify Postgres The client of . In our example ,K3s The client side. . therefore , I am for K3s Created a self signed certificate , And will ssl_ca_file Point to K3s Cluster self signed public certificate .

• restart Postgres Server to apply the new configuration ：

systemctl restart postgresql

We have successfully deployed Postgres database , Now let's turn to K3s Virtual machine and install there K3s.

K3s Server 1 virtual machine

• Install with the correct flags and values K3s Server：

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="postgres://K3s:[email protected]:5432/K3s" --datastore-cafile="/home/sles/postgres.crt" --token=K3s --datastore-certfile="/home/sles/K3s.crt" --datastore-keyfile="/home/sles/K3s.key" --tls-san=10.161.129.118

--datastore-endpoint：Postgres The format is postgres://username:password@hostname :port/database-name. In this example , I created a K3s role , The password is K3s, The database name is also K3s. I use the postgres.rancher.rke As hostname, Because the certificate is used CN Value is created as a name .

--datastore-cafile： Set to Postgres The public key certificate of , In order to K3s Use this certificate to verify Postgres. In a self signed certificate , Public certificates act as CA, You can verify yourself .

--datastore-certfile： This is the logo K3s The public certificate of the cluster .

--datastore-keyfile： Belong to K3s The private key of the cluster .

--token： Will create a Secret password , For other servers or Agent Connect to K3s colony .

--tls-san： Of the load balancer IP Address .

• In order to make K3s analysis postgres.rancher.rke2, I am here /``etc/hosts The following contents are attached at the end of the document ：

10.161.129.212  postgres.rancher.rke2

among 10.161.129.212 yes Postgres Server's IP Address .

• Now? , verification K3s Server Is it working properly and connected to Postgres：

export KUBECONFIG=/etc/rancher/k3s/k3s.yamlkubectl get pods -A

You should see all in running State of pod. without , You can run ：

journalctl -xe

This allows you to view the installation K3s Server Error occurred while .

K3s Server 2 virtual machine

• Use and install the first Server Use the same command to install the second K3s Server：

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="postgres://K3s:[email protected]:5432/K3s" --datastore-cafile="/home/sles/postgres.crt" --token=K3s --datastore-certfile="/home/sles/K3s.crt" --datastore-keyfile="/home/sles/K3s.key" --tls-san=10.161.129.118

• verification K3s Server Is it working properly and connected to Postgres：

export KUBECONFIG=/etc/rancher/k3s/k3s.yamlkubectl get pods -A

The result should be similar to the following ：

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGEkube-system   svclb-traefik-55frd                       2/2     Running   0          11mkube-system   svclb-traefik-x59vc                       2/2     Running   0          2m43skube-system   local-path-provisioner-6c79684f77-55tkc   1/1     Running   0          107skube-system   coredns-d76bd69b-5n8s7                    1/1     Running   0          107skube-system   traefik-df4ff85d6-88phx                   1/1     Running   0          107skube-system   metrics-server-7cd5fcb6b7-x7t2r           1/1     Running   0          107s

• You can also check whether there are two in the cluster by running this command Server, And see if they have master role ：

kubectl get nodes

The result should be similar to the following ：

NAME           STATUS   ROLES                  AGE   VERSIONk3s-server-1   Ready    control-plane,master   14m   v1.23.6+K3s1k3s-server-2   Ready    control-plane,master   29s   v1.23.6+K3s1

Connect to the scheduling workload K3s Agent Before , We need to be in Server Add a load balancer in front , For users or K3s Agent Communicate with .

Nginx Load balancer virtual machine

I'm on the network stack 4 Layer of K3s colony Server In front of the node Nginx. We will put the port 6443 All requests are forwarded to the load balancer , Then send one of them K3s Server.Kubernetes API server Will listen on this port .

• install Nginx package ：

zypper in nginx

• establish /etc/nginx/nginx.conf File and enter the following ：

load_module /usr/lib64/nginx/modules/ngx_stream_module.so;worker_processes 4;worker_rlimit_nofile 40000;events { worker_connections 8192;}stream { log_format logs '$remote_addr - - [$time_local] $protocol $status $bytes_sent $bytes_received $session_time "$upstream_addr"'; access_log /var/log/nginx/access.log logs; upstream K3s_api_server {  least_conn;  server 10.161.129.54:6443 max_fails=3 fail_timeout=5s;  server 10.161.129.154:6443 max_fails=3 fail_timeout=5s; } server {  listen 6443;  proxy_pass K3s_api_server; }

We use least_conn Algorithm to determine which request should be sent K3s Server.Nginx According to this algorithm, the request will be routed to the one with the least active Connected Server.

• restart Nginx For changes to take effect ：

nginx -s reloadsystemctl reload nginx && systemctl restart nginx

thus , We have set up the load balancer , So now anyone can communicate with our K3s Server signal communication . Now let's add a K3s Agent,Agent Will communicate with this load balancer and register Agent.

K3s Agent virtual machine

• Use the following command to install K3s Agent.

--server： Load Balancer IP Address .

curl -sfL https://get.k3s.io | sh -s - agent --token=K3s --server https://10.161.129.118:6443

K3s Server 1 virtual machine

• Now? , You can check by running this command K3s Agent Have you successfully registered ：

kubectl get nodes

The result should be similar to the following ：

NAME           STATUS   ROLES                  AGE    VERSIONk3s-server-1   Ready    control-plane,master   41m    v1.23.6+K3s1k3s-server-2   Ready    control-plane,master   28m    v1.23.6+K3s1k3s-agent      Ready    <none>                 105s   v1.23.6+K3s1

We have successfully installed Postgres、K3s Server And a K3s Agent. You now have an external database HA K3s colony . Please note that , Only deployed Postgres and Nginx Can be achieved completely HA. For more information , You can check the links in the reference .

Reference resources

• K3s Schema document ：https://rancher.com/docs/k3s/latest/en/architecture/#high-availability-with-an-external-db

• K3s Data storage documents ：https://rancher.com/docs/k3s/latest/en/installation/datastore/

• K3s Installation document ：https://rancher.com/docs/k3s/latest/en/installation/ha/

• Nginx Load balancer documentation ：https://rancher.com/docs/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nginx/