K8s set up redis cluster

Environmental preparation ：

Install well K8S, Not built K8S You can view ：kubeadm Way to build K8S colony 、RKE build K8S【 Strongly recommend 】

You need to know your intranet IP, For the back NFS and PV Configuration of

# View the intranet through the command IP, My network segment is 172.31.0.0/24
kubectl get nodes -o wide
#================================================================================================
NAME           STATUS   ROLES                      AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   
172.31.0.110   Ready    controlplane,etcd,worker   7d22h   v1.22.9   172.31.0.110   <none>         
172.31.0.115   Ready    worker                     7d22h   v1.22.9   172.31.0.115   <none>         
172.31.0.82    Ready    worker                     7d22h   v1.22.9   172.31.0.82    <none>        
#================================================================================================

1. establish NFS Storage

establish NFS Storage is mainly for Redis Provide stable back-end storage , When Redis Of Pod After restart or migration , You can still get the original data . here , Let's create NFS, And then by using PV by Redis Mount a remote NFS route .

yum -y install nfs-utils rpcbind（nfs-utils The main package provides the file system 、rpcbind Provide rpc agreement ）

newly added /etc/exports file , Used to set the path to be shared ：

# edit /etc/exports file , Add content 
vi /etc/exports
#================================================================================================
/usr/local/kubernetes/redis/pv1 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv2 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv3 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv4 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv5 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv6 172.31.0.0/24(rw,sync,no_root_squash)
#================================================================================================

Create the corresponding folder and authorize

mkdir -p /usr/local/kubernetes/redis/pv{
    1..6}
chmod 777 /usr/local/kubernetes/redis/pv{
    1..6}

Start the service

# start-up nfs and rpcbind service 
systemctl enable nfs 
systemctl enable rpcbind
systemctl start nfs
systemctl start rpcbind

exportfs -v
#================================================================================================
/usr/local/kubernetes/redis/pv1
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv2
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv3
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv4
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv5
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv6
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
#================================================================================================

All nodes are installed nfs client

# Install on all nodes 
yum -y install nfs-utils 

# Verify that the name of the directory mounted by the client is displayed 
showmount -e 172.31.0.110

2. establish PV【server by master Node's intranet IP】

vi pv.yaml 
#================================================================================================
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv1"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-vp2
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv2"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv3"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv4"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv5"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv6
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv6"
#================================================================================================
# Execution creation pv
kubectl create -f pv.yaml 

kubectl get pv

3. establish Configmap

# establish redis Folder and switch to folder 
mkdir /etc/redis && cd /etc/redis

# establish redis.conf file 
vi redis.conf
#================================================================================================
protected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
#================================================================================================

# take reids.conf To configMap file 
kubectl create configmap redis-conf --from-file=redis.conf

# View the created configmap:
kubectl describe cm redis-conf
#================================================================================================
Name:         redis-conf
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
redis.conf:
----
protected-mode noprotected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

Events:  <none>
#================================================================================================

4. establish headless service

Headless service yes StatefulSet The foundation of stable network identification

# establish headless-service.yaml  file 
vi headless-service.yaml 
#================================================================================================
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster
#================================================================================================
# establish 
kubectl create -f headless-service.yaml
# see 
kubectl get svc
# You can see , The service name is redis-service, Its CLUSTER-IP by None, That means this is one “ Headless ” service 
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.43.0.1    <none>        443/TCP    7d19h
redis-service   ClusterIP   None         <none>        6379/TCP   5s

5. establish Redis StatefulSet

# establish redis.yaml
vi redis.yaml 
#================================================================================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-app
spec:
  serviceName: "redis-service"
  replicas: 6 
  selector: 
    matchLabels:
      app: redis
      appCluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      containers:
      - name: redis
        image: "redis:6.0.6"
        command: ["/bin/bash", "-ce", "tail -f /dev/null"]
        command: ["redis-server"]
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"
            mountPath: "/var/lib/redis"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: "redis"
      resources:
        requests:
          storage: 1Gi

#================================================================================================
# Create new resources 
kubectl create -f redis.yaml
# see redis pods
kubectl get pods -o wide 
#================================================================================================
NAME          READY   STATUS    RESTARTS   AGE    IP           NODE           NOMINATED NODE   READINESS 
redis-app-0   1/1     Running   0          4m57s   10.42.1.31   172.31.0.115   <none>           <none>
redis-app-1   1/1     Running   0          4m56s   10.42.2.23   172.31.0.82    <none>           <none>
redis-app-2   1/1     Running   0          4m54s   10.42.0.21   172.31.0.110   <none>           <none>
redis-app-3   1/1     Running   0          4m51s   10.42.1.32   172.31.0.115   <none>           <none>
redis-app-4   1/1     Running   0          4m50s   10.42.2.24   172.31.0.82    <none>           <none>
redis-app-5   1/1     Running   0          4m48s   10.42.0.22   172.31.0.110   <none>           <none>
#================================================================================================
# see PV
kubectl get pv
#================================================================================================
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS 
nfs-pv1   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-2   redis     
nfs-pv3   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-0   redis     
nfs-pv4   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-4   redis     
nfs-pv5   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-5   redis     
nfs-pv6   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-1   redis     
nfs-vp2   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-3   redis    
#================================================================================================

6. initialization Redis colony

Initialize the cluster using redis-trib This tool , direct yum install redis-trib This software , Perform initialization

yum -y install redis-trib

redis-trib create --replicas 1 10.42.1.31:6379 10.42.2.23:6379 10.42.0.21:6379 10.42.1.32:6379 10.42.2.24:6379 10.42.0.22:6379

or docker Container building

docker pull inem0o/redis-trib

docker run --rm -ti inem0o/redis-trib create --replicas 1 10.42.1.31:6379 10.42.2.23:6379 10.42.0.21:6379 10.42.1.32:6379 10.42.2.24:6379 10.42.0.22:6379

Building a successful

#=============================================== Into the container =========================================
kubectl  exec  -it  redis-app-0  -- redis-cli  -c
#=============================================== View cluster =========================================
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:82
cluster_stats_messages_pong_sent:80
cluster_stats_messages_sent:162
cluster_stats_messages_ping_received:75
cluster_stats_messages_pong_received:82
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:162
#=============================================== Check out the characters =========================================
127.0.0.1:6379> role
1) "master"
2) (integer) 84
3) 1) 1) "10.42.1.36"
      2) "6379"
      3) "84"
#=============================================== Look at the node =========================================
127.0.0.1:6379> cluster nodes
fc63373625ab65f480c5343ffca4f7d650af8ee6 10.42.0.25:[email protected] master - 0 1653709741112 3 connected 10923-16383
c444518310f7be61ace5895605f24fe23fd4625b 10.42.2.28:[email protected] slave 635a262b8e0b2e340acf68262a73870099f40be6 0 1653709740000 2 connected
e57fff1006a56eea193179f0ffcf91ee91e8b800 10.42.1.36:[email protected] slave 1fe5682e876057cc1e2346b4434008abb7ed2ab8 0 1653709740512 1 connected
635a262b8e0b2e340acf68262a73870099f40be6 10.42.2.27:[email protected] master - 0 1653709741000 2 connected 5461-10922
289c8aae1b7093b3ae58e7be9bc6bc75abd8dcef 10.42.0.26:[email protected] slave fc63373625ab65f480c5343ffca4f7d650af8ee6 0 1653709740612 3 connected
1fe5682e876057cc1e2346b4434008abb7ed2ab8 10.42.1.35:[email protected] myself,master - 0 1653709739000 1 connected 0-5460

7、 Create for accessing Service

vi redis-access-service.yaml 

apiVersion: v1
kind: Service
metadata:
  name: redis-access-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
    nodePort: 6379
  selector:
    app: redis
    appCluster: redis-cluster
  type: NodePort
    

kubectl get svc redis-access-service -o wide

Be careful ： What I'm using here is NodePort Directly open the external port for use , Want to use the default ClusterId, And then through Ingress Open ports for access .