Read how to deploy highly available k3s with external database

Introduce

Are you deploying in high availability mode  Kubernetes  And encountered difficulties in configuring the remote backup database ？ This article will show you how to have an external database  Postgres  Of  HA  Deployed in configuration  K3s.

K3s（https://k3s.io/） It is used for authentication of Internet of things and edge computing  Kubernetes  Distribution version . I deployed it in  IBM Z  On the virtual machine in the mainframe . I have no choice  etcd, It USES  Postgres  As  K3s  Storage scheme of cluster . I blame  HA  The pattern is deployed  Postgres. Want to know how to  HA  Deploy in mode  Postgres  Information about , Please refer to the official documentation ：https://www.postgresql.org/docs/13/high-availability.html

Postgres  and  K3s  The communication between clusters is  SSL  Safe . I am here  K3s server  The first one  4  Layer used  Nginx  Load balancing . We may need to add new server nodes or shut down the server in the future . therefore , If not used  K3s Server IP  Instead, use a load balancer , We can avoid this kind of trouble , It can also be realized  HA. Please note that ,Nginx  The load balancer may also become a single point of failure . About in  HA  Deploy in mode  Nginx, See documentation ：https://www.nginx.com/products/nginx/high-availability/

HA  To configure

The following figure describes me in  HA  The lower part of the mode deploys  K3s  The way of clustering . Even one  Server  Something goes wrong , another  Server  Still accessible , And the load balancer can route all requests to accessible  Server. because  K3s  All components of are stateless , Even if  Server  Something goes wrong , We will not lose any information . All information is stored in  Postgres  in , And must be  HA.

With load balancer and database  K3s HA  Pattern

Environmental Science

Before we start , You need the following environment ：

operating system  - SLE15SP2  virtual machine  - 5 VCPU - 2 RAM - 8 GB  disk  - 30 GB Arch - s390x

Here are the of each virtual machine I use  IP  Address , You can check these at any time in this article  IP  The meaning of address ：

K3s Server 1 - 

10.161.129.54

K3s Server 2 - 

10.161.129.154

K3s Agent - 

10.161.129.196

Postgres - 

10.161.129.212

Nginx - 

10.161.129.118

I'm on the mainframe  (s390x)  Deployment in virtual machine , You can also use the same commands on any other architecture . Now? , Let's take a look at the installation  Postgres  and  K3s  Cluster command .

install  Postgres

I installed  postgres10. We name this virtual machine  Postgres. You can copy and paste the following commands to install  Postgres. For safety's sake , Please make sure  Postgres  Database and  K3s  There are interactions between clusters  TLS  signal communication . I used a self signed certificate to identify the use of  OpenSSL  Created  Postgres.

Postgres  virtual machine

• Use  zypper  install  Postgres  package ：

zypper -n in postgresql10 postgresql10-server
systemctl start postgresql

• establish  K3s  database 、 User role and grant all access rights to user role ：

sudo -u postgres psql
create database K3s;
create user K3s with encrypted password 'K3s';
grant all privileges on database K3s to K3s;
exit;

We will use  

K3s

  Database to store cluster information . We use  

K3s

  Users come for  K3s  Cluster  Postgres  Database authentication .

• Create logo  Postgres  The self signed certificates of the server and store them in  
  /var/lib/pgsql/data/
    in ：

openssl req -new -x509 -days 365 -nodes -text -out /var/lib/pgsql/data/postgres.crt -keyout /var/lib/pgsql/data/postgres.key -subj "/CN=postgres.rancher.rke2" -addext "subjectAltName=DNS:postgres.rancher.rke2"

• Ensure that access to the private key is restricted ：

chmod 0600 /var/lib/pgsql/data/postgres.key
chown postgres:postgres /var/lib/pgsql/data/postgres.key

• Mark  Postgres  Copy the public key certificate of to two  K3s Server, send  K3s Server  Can verify  Postgres  To carry out  SSL  signal communication ：

scp /var/lib/pgsql/data/postgres.crt [email protected]:
scp /var/lib/pgsql/data/postgres.crt [email protected]:

• take  
  /var/lib/pgsql/data/pg_hba.conf
    Replace with the following ：

# TYPE DATABASE USER ADDRESS METHOD

# "local" is for Unix domain socket connections onlyf
local all all peer
# IPv4 local connections:
host all all 127.0.0.1/32 ident
hostssl all all 0.0.0.0/0 md5 clientcert=verify-full

The contents of this document indicate ,localhost  No password or  SSL  Communication can connect to the database . however ,Postgres  All other connections must be through  SSL  Client authentication （K3s Server  yes  Postgres  The client of ） And password authentication to communicate .

In further setting  Postgres  Before the database , Let's create the logo first  K3s  Cluster the certificate and copy the certificate to  Postgres  virtual machine , In order to  Postgres  verification  K3s Server.

K3s Server 1  virtual machine

• Create a logo  K3s  Cluster self signed certificate , And grant private key permission ：

openssl req -new -x509 -days 365 -nodes -text -out K3s.crt -keyout K3s.key -subj "/CN=K3s" -addext "subjectAltName=DNS:K3s"
chmod 0600 K3s.key

• Copy the public key certificate to  Postgres  host , In order to  Postgres  Can verify  K3s  client ：

scp /home/sles/K3s.crt [email protected]:

• Copy public and private keys to other  K3s Server. Two sets of  Server  form  K3s  colony ：

scp /home/sles/K3s.crt /home/sles/K3s.key [email protected]:

Next , Let's keep looking  Postgres  virtual machine .

Postgres  virtual machine

• take  K3s.crt  Move to  
  /var/lib/pgsql/data
    The catalogue is for  Postgres  Configuration file usage ：

mv /home/sles/K3s.crt /var/lib/pgsql/data/

• take  
  /var/lib/pgsql/data/postgresql.conf
    The content of is modified to the following value ：

listen_addresses = '*'
ssl = on
#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on
#ssl_ecdh_curve = 'prime256v1'
#ssl_dh_params_file = ''
ssl_cert_file = '/var/lib/pgsql/data/postgres.crt'
ssl_key_file = '/var/lib/pgsql/data/postgres.key'
ssl_ca_file = '/var/lib/pgsql/data/K3s.crt'

listen_addresses

： Set to  

*

  or  Postgres  Server's  IP. This ensures that  Postgres  The server can listen to nodes  IP  Address .

ssl

： open  SSL  To communicate only in a secure way .

ssl_cert_file

 

ssl_key_file

： identification  Postgres  Certificate of database . I have created the certificate at the beginning of this article , Now just point them to the certificate location .

ssl_ca_file

： This is a  CA（ Certification authority ） certificate , Used to identify  Postgres  The client of . In our example ,K3s  The client side. . therefore , I am for  K3s  Created a self signed certificate , And will  

ssl_ca_file

  Point to  K3s  Cluster self signed public certificate .

• restart  Postgres  Server to apply the new configuration ：

systemctl restart postgresql

We have successfully deployed  Postgres  database , Now let's turn to  K3s  Virtual machine and install there  K3s.

K3s Server 1  virtual machine

• Install with the correct flags and values  K3s Server：

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="postgres://K3s:[email protected]:5432/K3s" --datastore-cafile="/home/sles/postgres.crt" --token=K3s --datastore-certfile="/home/sles/K3s.crt" --datastore-keyfile="/home/sles/K3s.key" --tls-san=10.161.129.118

--datastore-endpoint

：Postgres  The format is  

postgres://username:[email protected]:port/database-name

. In this example , I created a  

K3s

  role , The password is  

K3s

, The database name is also  

K3s

. I use the  

postgres.rancher.rke

  As  hostname, Because the certificate is used  CN  Value is created as a name .

--datastore-cafile

： Set to  Postgres  The public key certificate of , In order to  K3s  Use this certificate to verify  Postgres. In a self signed certificate , Public certificates act as  CA, You can verify yourself .

--datastore-certfile

： This is the logo  K3s  The public certificate of the cluster .

--datastore-keyfile

： Belong to  K3s  The private key of the cluster .

--token

： Will create a  Secret  password , For other servers or  Agent  Connect to  K3s  colony .

--tls-san

： Of the load balancer  IP  Address .

• In order to make  K3s  analysis  
  postgres.rancher.rke2
  , I am here  
  /``etc/hosts
    The following contents are attached at the end of the document ：

10.161.129.212 postgres.rancher.rke2

among  

10.161.129.212

  yes  Postgres  Server's  IP  Address .

• Now? , verification  K3s Server  Is it working properly and connected to  Postgres：

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pods -A

You should see all in  

running

  State of  pod. without , You can run ：

journalctl -xe

This allows you to view the installation  K3s Server  Error occurred while .

K3s Server 2  virtual machine

• Use and install the first  Server  Use the same command to install the second  K3s Server：

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="postgres://K3s:[email protected]:5432/K3s" --datastore-cafile="/home/sles/postgres.crt" --token=K3s --datastore-certfile="/home/sles/K3s.crt" --datastore-keyfile="/home/sles/K3s.key" --tls-san=10.161.129.118

• verification  K3s Server  Is it working properly and connected to  Postgres：

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pods -A

The result should be similar to the following ：

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system svclb-traefik-55frd 2/2 Running 0 11m
kube-system svclb-traefik-x59vc 2/2 Running 0 2m43s
kube-system local-path-provisioner-6c79684f77-55tkc 1/1 Running 0 107s
kube-system coredns-d76bd69b-5n8s7 1/1 Running 0 107s
kube-system traefik-df4ff85d6-88phx 1/1 Running 0 107s
kube-system metrics-server-7cd5fcb6b7-x7t2r 1/1 Running 0 107s

• You can also check whether there are two in the cluster by running this command  Server, And see if they have  
  master
    role ：

kubectl get nodes

The result should be similar to the following ：

NAME STATUS ROLES AGE VERSION
k3s-server-1 Ready control-plane,master 14m v1.23.6+K3s1
k3s-server-2 Ready control-plane,master 29s v1.23.6+K3s1

Connect to the scheduling workload  K3s Agent  Before , We need to be in  Server  Add a load balancer in front , For users or  K3s Agent  Communicate with .

Nginx  Load balancer virtual machine

I'm on the network stack  4  Layer of  K3s  colony  Server  In front of the node  Nginx. We will put the port  6443  All requests are forwarded to the load balancer , Then send one of them  K3s Server.Kubernetes API server  Will listen on this port .

• install  Nginx  package ：

zypper in nginx

• establish  
  /etc/nginx/nginx.conf
    File and enter the following ：

load_module /usr/lib64/nginx/modules/ngx_stream_module.so;

worker_processes 4;
worker_rlimit_nofile 40000;


events {
 worker_connections 8192;
}

stream {
 log_format logs '$remote_addr - - [$time_local] $protocol $status $bytes_sent $bytes_received $session_time "$upstream_addr"';

 access_log /var/log/nginx/access.log logs;

 upstream K3s_api_server {
 least_conn;
 server 10.161.129.54:6443 max_fails=3 fail_timeout=5s;
 server 10.161.129.154:6443 max_fails=3 fail_timeout=5s;
 }
 server {
 listen 6443;
 proxy_pass K3s_api_server;
 }

We use  

least_conn

  Algorithm to determine which request should be sent  K3s Server.Nginx  According to this algorithm, the request will be routed to the one with the least  active  Connected  Server.

• restart  Nginx  For changes to take effect ：

nginx -s reload
systemctl reload nginx && systemctl restart nginx

thus , We have set up the load balancer , So now anyone can communicate with our  K3s Server  signal communication . Now let's add a  K3s Agent,Agent  Will communicate with this load balancer and register  Agent.

K3s Agent  virtual machine

• Use the following command to install  K3s Agent.

--server

： Load Balancer  IP  Address .

curl -sfL https://get.k3s.io | sh -s - agent --token=K3s --server https://10.161.129.118:6443

K3s Server 1  virtual machine

• Now? , You can check by running this command  K3s Agent  Have you successfully registered ：

kubectl get nodes

The result should be similar to the following ：

NAME STATUS ROLES AGE VERSION
k3s-server-1 Ready control-plane,master 41m v1.23.6+K3s1
k3s-server-2 Ready control-plane,master 28m v1.23.6+K3s1
k3s-agent Ready <none> 105s v1.23.6+K3s1

We have successfully installed  Postgres、K3s Server  And a  K3s Agent. You now have an external database  HA K3s  colony . Please note that , Only deployed  Postgres  and  Nginx  Can be achieved completely  HA. For more information , You can check the links in the reference .

Reference resources

• K3s  Schema document ：https://rancher.com/docs/k3s/latest/en/architecture/#high-availability-with-an-external-db

• K3s  Data storage documents ：https://rancher.com/docs/k3s/latest/en/installation/datastore/

• K3s  Installation document ：https://rancher.com/docs/k3s/latest/en/installation/ha/

• Nginx  Load balancer documentation ：https://rancher.com/docs/rancher/v2.5/en/installation/resources/k8s-tutorials/infrastructure-tutorials/nginx/