当前位置：网站首页>Deploy MySQL based on statefulset in kubernetes (Part 2)

Deploy MySQL based on statefulset in kubernetes (Part 2)

2022-06-10 08:57:00 【Kubesphere cloud native】

Hello everyone , I'm old Z！

The last article realized MySQL Database is based on KubeSphere The deployment of K8s Installation and deployment on the cluster , The deployment mode adopts the form of graphical interface . This article will show you how to use GitOps To deploy MySQL, All involved in the deployment process YAML Files will use Git Version management , And stored in Git Warehouse . therefore , This article will also cover GitOps Basic operation of .

Native K8s Use GitOps Deploy MySQL

In the last article, we passed KubeSphere Deploy a single instance MySQL, So original K8s And how to operate ？GitOps What is it again? 、 How to realize ？

What is? GitOps( Excerpt from online article )

  • GitOps It's a set of uses Git To manage infrastructure and application configuration practices , and Git It refers to an open source control system .
  • GitOps In the process of operation, use Git A single source of facts for declarative infrastructure and applications .
  • GitOps Use Git Pull requests to automatically manage infrastructure provisioning and deployment .
  • Git The repository contains all the states of the system , Therefore, the change trace of system status can be viewed or audited .
  • GitOps Often used as K8s And the operation and maintenance mode of cloud native application development , And it can realize to K8s Continuous deployment of .
  • GitOps Is a way of continuous delivery . Its core idea is to store the declarative infrastructure and applications of the application system in Git In the version Library .

Prepare resource allocation list - Train of thought

We know how to play K8s The essential skill of is to write the resource allocation list , In general use YAML Format file to create our expected resource configuration .

At this time, we should also write by hand MySQL Resource allocation list of ？ I'm in a panic , I can't remember all the parameters .

NO！NO！NO！ It's time for opportunism , The key to selling in front has been opened here .

We have already passed KubeSphere The graphical interface created by MySQL Resource allocation of , and KubeSphere A great feature is the ability to edit resources directly online YAML file .

We can create resources , Direct editing YAML File creation resources . It can also be edited YAML To modify existing resources .

Of course. , You don't use a graphical interface , Directly in K8s The bottom layer uses the command line to get YAML Format output , Re edit , It's OK, too .

Sort out MySQL The resources included in the resource configuration list .

  • StatefulSet( Stateful replica set )
  • Service( service )
    • Within a cluster （Headless）
    • Outside the cluster （ Custom service ）
  • ConfigMap
  • Secret

Next, we will get the resource configuration lists respectively .

Prepare resource allocation list

ConfigMap

To configure -> Configuration Dictionary , find mysql-cnf, Click on the right Three vertical points , Click on edit YAML.

open edit YAML page , You can copy everything directly , You can also click the download icon in the upper right corner , Download the file ( You can also use the upload icon to upload files ).

The obtained current network configuration cannot be completely used , Need modification , Clean up some metadata information automatically added by the system .

Current network mysql-cfm.yaml.

kind: ConfigMap
apiVersion: v1
metadata:
  name: mysql-cnf
  namespace: lstack
  annotations:
    kubesphere.io/creator: lstack
data:
  custom.cnf: |-
    [mysqld]
    #performance setttings
    lock_wait_timeout = 3600
    open_files_limit    = 65535
    back_log = 1024
    max_connections = 512
    max_connect_errors = 1000000
    table_open_cache = 1024
    table_definition_cache = 1024
    thread_stack = 512K
    sort_buffer_size = 4M
    join_buffer_size = 4M
    read_buffer_size = 8M
    read_rnd_buffer_size = 4M
    bulk_insert_buffer_size = 64M
    thread_cache_size = 768
    interactive_timeout = 600
    wait_timeout = 600
    tmp_table_size = 32M
    max_heap_table_size = 32M

The modified mysql-cfm.yaml.

kind: ConfigMap
apiVersion: v1
metadata:
  name: mysql-cnf
  namespace: lstack
data:
  custom.cnf: |-
    [mysqld]
    #performance setttings
    lock_wait_timeout = 3600
    open_files_limit    = 65535
    back_log = 1024
    max_connections = 512
    max_connect_errors = 1000000
    table_open_cache = 1024
    table_definition_cache = 1024
    thread_stack = 512K
    sort_buffer_size = 4M
    join_buffer_size = 4M
    read_buffer_size = 8M
    read_rnd_buffer_size = 4M
    bulk_insert_buffer_size = 64M
    thread_cache_size = 768
    interactive_timeout = 600
    wait_timeout = 600
    tmp_table_size = 32M
    max_heap_table_size = 32M

Secret

To configure -> Confidential dictionary , find mysql-secret, Click on the right Three vertical points , Click on edit YAML.

Current network mysql-secret.yaml.

kind: Secret
apiVersion: v1
metadata:
  name: mysql-secret
  namespace: lstack
  annotations:
    kubesphere.io/creator: lstack
data:
  MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
type: Opaque

The modified mysql-secret.yaml.

kind: Secret
apiVersion: v1
metadata:
  name: mysql-secret
  namespace: lstack
data:
  MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
type: Opaque

Here's a word ,Secret The value in is in base64 It's encrypted , So here MYSQL_ROOT_PASSWORD, Use the actual password base64 The way to encrypt .

StatefulSet

Application load -> The workload -> Stateful replica set , find mysql, Click on the right Three vertical points , Click on edit YAML.

Current network mysql-sts.yaml.

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mysql
  namespace: lstack
  labels:
    app: mysql
  annotations:
    kubesphere.io/creator: lstack
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: mysql
      annotations:
        logging.kubesphere.io/logsidecar-config: '{}'
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
        - name: volume-rca2zx
          configMap:
            name: mysql-cnf
            items:
              - key: custom.cnf
                path: custom.cnf
            defaultMode: 420
      containers:
        - name: lstack-mysql
          image: 'mysql:5.7.38'
          ports:
            - name: tcp-mysql
              containerPort: 3306
              protocol: TCP
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: MYSQL_ROOT_PASSWORD
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 500m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              mountPath: /etc/localtime
            - name: data
              mountPath: /var/lib/mysql
            - name: volume-rca2zx
              readOnly: true
              mountPath: /etc/mysql/conf.d/custom.cnf
              subPath: custom.cnf
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: default
      serviceAccount: default
      securityContext: {}
      schedulerName: default-scheduler
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: data
        namespace: lstack
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: glusterfs
        volumeMode: Filesystem
      status:
        phase: Pending
  serviceName: mysql-1dpr
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 10

The modified mysql-sts.yaml.

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mysql
  namespace: lstack
  labels:
    app: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
        - name: volume-cnf
          configMap:
            name: mysql-cnf
            items:
              - key: custom.cnf
                path: custom.cnf
            defaultMode: 420
      containers:
        - name: lstack-mysql
          image: 'mysql:5.7.38'
          ports:
            - name: tcp-mysql
              containerPort: 3306
              protocol: TCP
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: MYSQL_ROOT_PASSWORD
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 500m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              mountPath: /etc/localtime
            - name: data
              mountPath: /var/lib/mysql
            - name: volume-cnf
              mountPath: /etc/mysql/conf.d/custom.cnf
              subPath: custom.cnf
  volumeClaimTemplates:
    - metadata:
        name: data
        namespace: lstack
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: glusterfs
  serviceName: mysql-headless

Service

First create Headless service , Application load -> service ->, find mysql-xxxx(mysql), Click on the right Three vertical points , Click on edit YAML.

Current network mysql-headless.yaml.

kind: Service
apiVersion: v1
metadata:
  name: mysql-1dpr
  namespace: lstack
  labels:
    app: mysql
  annotations:
    kubesphere.io/alias-name: mysql
    kubesphere.io/creator: lstack
    kubesphere.io/serviceType: statefulservice
spec:
  ports:
    - name: tcp-mysql
      protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: mysql
  clusterIP: None
  clusterIPs:
    - None
  type: ClusterIP
  sessionAffinity: None
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack

The modified mysql-headless.yaml.

kind: Service
apiVersion: v1
metadata:
  name: mysql-headless
  namespace: lstack
  labels:
    app: mysql
spec:
  ports:
    - name: tcp-mysql
      protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: mysql
  clusterIP: None
  type: ClusterIP

Look at the custom mysql-external service , Application load -> service ->, find mysql-external, Click on the right Three vertical points , Click on edit YAML.

Current network mysql-external.yaml.

kind: Service
apiVersion: v1
metadata:
  name: mysql-external
  namespace: lstack
  labels:
    app: mysql-external
  annotations:
    kubesphere.io/creator: lstack
spec:
  ports:
    - name: tcp-mysql-external
      protocol: TCP
      port: 3306
      targetPort: 3306
      nodePort: 32529
  selector:
    app: mysql
  clusterIP: 10.233.36.71
  clusterIPs:
    - 10.233.36.71
  type: NodePort
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack

Here's a point to make nodePort This parameter , If K8s Cluster controllable , It is recommended to plan a set of service port usage specifications , Every need nodePort All services specify fixed ports , This is conducive to the standardization of operation and maintenance .

The modified mysql-external.yaml( Be careful nodePort Parameter is not specified ).

kind: Service
apiVersion: v1
metadata:
  name: mysql-external
  namespace: lstack
  labels:
    app: mysql-external
spec:
  ports:
    - name: tcp-mysql-external
      protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: mysql
  type: NodePort

take MySQL Resource configuration list submitted to Git Warehouse .

By the above operation , We got MySQL Resource allocation list of .

I have OCD , Like to store by category , So I used it 4 File ,mysql-headless.yaml Follow mysql-sts.yaml Merge into one file. Of course, you can also put it into a configuration file .

  • mysql-external.yaml
  • mysql-sts.yaml
  • mysql-secret.yaml
  • mysql-cfm.yaml

Submit the resource configuration list to Git Warehouse

choice GitHub As the main warehouse ,Gitee As a synchronous warehouse ( artificial ).

All of the documents in this series k8s The resource configuration manifest file for uses a public repository , The production environment suggests creating a configuration repository for each service , It is conducive to more refined version control .

This article is to demonstrate the use of the primary and standby warehouses , All the choices Github and Gitee Two kinds of Git service , In actual use, it is recommended to choose... For better use experience Gitee.

stay GitHub Build a new warehouse , Warehouse name k8s-yaml, Add one README File initialization Repository , Click on Create repository, Confirm creation .

Code warehouse Clone Back home .

$ git clone [email protected]:devops/k8s-yaml.git
$ ls k8s-yaml 
README.md

Create a new folder , Use your favorite text editor ( recommend vscode) edit MySQL Resource allocation list of , And put the file into the newly created folder .

For future extensibility , We created one here single Named secondary directory , A resource configuration manifest file that stores a single instance .

$ mkdir -p k8s-yaml/mysql/single
$ ls -l k8s-yaml/mysql/single
total 32
-rw-r--r--  1 z  staff   646  5 11 19:23 mysql-cfm.yaml
-rw-r--r--  1 z  staff   266  5 11 19:31 mysql-external.yaml
-rw-r--r--  1 z  staff   134  5 11 19:23 mysql-secret.yaml
-rw-r--r--  1 z  staff  1911  5 11 19:31 mysql-sts.yaml

The edited resource configuration file list , Submitted to the GitHub.

$ cd k8s-yaml
$ git add .
$ git commit -am ' add to MySQL single Resource allocation list '
$ git push

stay GitHub Check out , Confirm whether the code is submitted .

Next, synchronize the resource configuration list to Gitee Backup warehouse .

  • This paper adopts the method of manual push synchronization ( Personal habits )
  • Gitee It also supports automatic synchronization GitHub The warehouse of ( More convenient )

stay Gitee Build a new warehouse , Warehouse name k8s-yaml, Type default private , Click on establish .

After creation, it can be modified to open source in the warehouse settings .

After creation , Because when we created it , The configuration of initialization warehouse is not selected , therefore , A help page is displayed by default , Tell you how to submit code to the repository .

because , We already have a code warehouse , So we choose There are warehouses Configuration method of , Submit the existing code to Gitee.

Follow the help tips , it is to be noted that origin We'll change to gitee.

$ git remote add gitee https://gitee.com/zdevops/k8s-yaml.git
$ git push -u gitee

stay Gitee Check out , Confirm whether the code is submitted .

modify Gitee The warehouse is open source ( Optional ).

Gitee Warehouse -> management -> The warehouse is set -> essential information , Back most Open source or not , choice Open source , Warehouse publicity notice , Check all three , Click on preservation .

After modification , Your code warehouse is open source , Everyone can see it .

GitOps First experience - stay K8s Deployment on Cluster MySQL

MySQL The resource configuration list has been stored in Git Online warehouse , Next, open our GitOps Experience the journey .

Sign in k8s Of master node , Perform the following operation tasks .

In the production environment, it is recommended to create an independent operation and maintenance management node to manage the entire cluster , You can refer to 《 be based on KubeSphere Get along well with k8s- Operation and maintenance management node creation notes 》

install Git.

$ yum install git -y

establish devops Catalog , I choose /opt Catalog as devops Root directory .

$ mkdir /opt/devops
$ cd /opt/devops/

from Gitee download k8s-yaml Warehouse code .

$ git clone https://gitee.com/zdevops/k8s-yaml.git
$ ls k8s-yaml/
mysql  README.md

Because it is the same test environment , Clean up the existing MySQL service .

$ kubectl get secrets -n lstack 
NAME                  TYPE                                  DATA   AGE
default-token-x2gzv   kubernetes.io/service-account-token   3      31d
mysql-secret          Opaque                                1      2d20h

$ kubectl get configmaps -n lstack 
NAME               DATA   AGE
kube-root-ca.crt   1      31d
mysql-cnf          1      47h

$ kubectl get service -n lstack 
NAME                                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
glusterfs-dynamic-afe88cf4-86b1-4215-833a-534c5f779a22   ClusterIP   10.233.13.188   <none>        1/TCP            2d
mysql-1dpr                                               ClusterIP   None            <none>        3306/TCP         2d
mysql-external                                           NodePort    10.233.36.71    <none>        3306:32529/TCP   47h

$ kubectl get statefulsets -n lstack 
NAME    READY   AGE
mysql   1/1     2d

#  clear 
$ kubectl delete statefulsets mysql -n lstack
statefulset.apps "mysql" deleted
$ kubectl delete service mysql-external -n lstack
service "mysql-external" deleted
$ kubectl delete service mysql-1dpr -n lstack
service "mysql-1dpr" deleted
$ kubectl delete secrets mysql-secret -n lstack
secret "mysql-secret" deleted
$ kubectl delete configmaps mysql-cnf -n lstack
configmap "mysql-cnf" deleted

Use the resource configuration list to deploy with one click MySQL.

$ cd /opt/devops/k8s-yaml/
$ ls
mysql  README.md

$ kubectl apply -f mysql/single/

The verification results , Find out StatefulSet Not created , To analyze problems .

$ kubectl get statefulsets -n lstack
No resources found in lstack namespace.

#  At first I thought I had missed the configuration file ,ls Take a look at , I found that all the files were 
$ ls
mysql  README.md
$ cd mysql/
$ ls
single
$ cd single/
$ ls
mysql-cfm.yaml  mysql-external.yaml  mysql-secret.yaml  mysql-sts.yaml

#  Confirm the contents of the document , It is found that the file also has contents 
$ vi mysql-sts.yaml

#  Re execution , Found a clue , Why only service/mysql-headless  Resource allocation of , No, statefulset
$ kubectl apply -f mysql-sts.yaml 
service/mysql-headless unchanged

#  To confirm again , I found that I missed a little while editing the file , When a configuration file has multiple resource definitions , The configuration of different resources directly requires "---" Separate . Modify the configuration file and execute again , Found execution successful .
$ vi mysql-sts.yaml
$ cd ..

$ kubectl apply -f single/

$ kubectl get statefulsets -n lstack -o wide
NAME    READY   AGE   CONTAINERS     IMAGES
mysql   1/1     31s   lstack-mysql   mysql:5.7.38
$ kubectl get pods -n lstack -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE   READINESS GATES
mysql-0   1/1     Running   0          35s   10.233.116.59   ks-k8s-master-2   <none>           <none>

Back to our KubeSphere Management console for , Find out mysql The workload can also be displayed in the interface , This also verifies that in native k8s The operation on the will also directly reflect KubeSphere Management console for .

Second experience GitOps

Just by virtue of the above problems , Have a second experience GitOps. We directly modified the deployment server mysql-sts.yaml, And the modified results are verified successfully .

To demonstrate GitOps More scenes , Modify directly on the deployment server , Then submit to the online code repository .

In my actual work, I modify it on my office computer , Submit to the online code repository , Then the deployment server pulls the updated code .

The modified mysql-sts.yaml, Due to the space problem, only the key parts are demonstrated here ,StatefulSet The complete configuration of is shown in Gitee Warehouse or above .

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mysql
  namespace: lstack
  labels:
    app: mysql
...

---
kind: Service
apiVersion: v1
metadata:
  name: mysql-headless
  namespace: lstack
  labels:
    app: mysql
spec:
  ports:
    - name: tcp-mysql
      protocol: TCP
      port: 3306
      targetPort: 3306
  selector:
    app: mysql
  clusterIP: None
  type: ClusterIP

Submit the modified code to the code warehouse .

#  Check after modification git Warehouse changes 
$ git diff 
diff --git a/mysql/single/mysql-sts.yaml b/mysql/single/mysql-sts.yaml
index f775920..1eded9c 100644
--- a/mysql/single/mysql-sts.yaml
+++ b/mysql/single/mysql-sts.yaml
@@ -1,3 +1,4 @@
+---
 kind: StatefulSet
 apiVersion: apps/v1
 metadata:
@@ -68,6 +69,7 @@ spec:
         storageClassName: glusterfs
   serviceName: mysql-headless

+---
 kind: Service
 apiVersion: v1
 metadata:

#  Commit code changes locally 
$ git commit -am ' Repair mysql statefulset Configuration does not work '

# push To the online code warehouse , There is one warning You can ignore , You can also follow the prompts 
$ git push

see Gitee Is there any change in the online code warehouse .

On a personal office computer , Synchronize the updated code .

#  Update code 
$ git pull

#  Synchronize the updated code to Github
$ git push -u origin

see GitHub Is there any change in the online code warehouse .

Experience it again GitOps

Simulate a business scenario , Try it again GitOps.

  • MySQL After running online , Due to the increase in business volume , In the initial configuration parameters max_connections Is too small , Need to increase .

  • After adjustment of configuration parameters , Update online configuration , And restart the service ( Do not restart the production environment database easily , This need can be addressed with temporary modifications ).

  • Here is just a simple example , Let us experience GitOps, All configuration files in actual use are recommended to use Git Version control .

Edit local Git Warehouse MySQL In the resource configuration list mysql-cfm.yaml file , modify max_connections, from 512 become 1024.

Submit changes to Git Online warehouse .

#  Submit local changes 
$ git commit -am ' modify mysql-cnf in max_connections Value '

#  Submitted to the Github
$ git push

#  Synchronize to Gitee
$ git push -u gitee

Log in to the O & M management node , to update Git Code , And re run .

$ git pull

$ kubectl apply -f mysql/single/

#  see ConfigMap The change of 
$ kubectl get configmaps mysql-cnf -n lstack -o yaml
apiVersion: v1
data:
  custom.cnf: |-
    [mysqld]
    #performance setttings
    lock_wait_timeout = 3600
    open_files_limit    = 65535
    back_log = 1024
    max_connections = 1024
    max_connect_errors = 1000000
    table_open_cache = 1024
    table_definition_cache = 1024
    thread_stack = 512K
    sort_buffer_size = 4M
    join_buffer_size = 4M
    read_buffer_size = 8M
    read_rnd_buffer_size = 4M
    bulk_insert_buffer_size = 64M
    thread_cache_size = 768
    interactive_timeout = 600
    wait_timeout = 600
    tmp_table_size = 32M
    max_heap_table_size = 32M
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"custom.cnf":"[mysqld]\n#performance setttings\nlock_wait_timeout = 3600\nopen_files_limit    = 65535\nback_log = 1024\nmax_connections = 1024\nmax_connect_errors = 1000000\ntable_open_cache = 1024\ntable_definition_cache = 1024\nthread_stack = 512K\nsort_buffer_size = 4M\njoin_buffer_size = 4M\nread_buffer_size = 8M\nread_rnd_buffer_size = 4M\nbulk_insert_buffer_size = 64M\nthread_cache_size = 768\ninteractive_timeout = 600\nwait_timeout = 600\ntmp_table_size = 32M\nmax_heap_table_size = 32M"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"mysql-cnf","namespace":"lstack"}}
  creationTimestamp: "2022-05-12T07:20:07Z"
  name: mysql-cnf
  namespace: lstack
  resourceVersion: "8928391"
  uid: 1b7322cf-f11e-445d-a2ba-b42a90ade469

#  restart mysql pod Make configuration effective 
$ kubectl delete -f mysql/single/mysql-sts.yaml 

$ kubectl apply -f mysql/single/mysql-sts.yaml 

#  see mysql Whether the internal configuration of the container is updated 
$ kubectl exec  mysql-0  -n lstack -- cat /etc/mysql/conf.d/custom.cnf
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit    = 65535
back_log = 1024
max_connections = 1024
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32M

Bear in mind ！ The above example is just for you to experience GitOps, Do not restart the database server easily in the production environment , Unless you know what you're doing .

Now it has been verified , our MySQL The configuration of is available and stable , We record this good state , To avoid future changes that would corrupt , The original correct configuration cannot be found .

Give the current on our PC Git The code is marked with Tag, Record the current state ( It can also be operated through the management interface of the online warehouse ).

#  hit tag -a tag name  -m tag describe 
$ git tag -a v0.1  -m 'mysql version v0.1'

#  View existing tag
$ git tag -l
v0.1

#  see tag Details 
$ git show v0.1
tag v0.1
Tagger: devops <[email protected]>
Date:   Thu May 12 18:15:34 2022 +0800

mysql version v0.1

commit 180f97ac96da504a0b46eb4871ef423f64fde093 (HEAD -> main, tag: v0.1, origin/main, origin/HEAD, gitee/main)
Author: devops <[email protected]>
Date:   Thu May 12 17:48:18 2022 +0800

     modify mysql-cnf in max_connections Value 

diff --git a/mysql/single/mysql-cfm.yaml b/mysql/single/mysql-cfm.yaml
index e24d96d..50d1778 100644
--- a/mysql/single/mysql-cfm.yaml
+++ b/mysql/single/mysql-cfm.yaml
@@ -10,7 +10,7 @@ data:
     lock_wait_timeout = 3600
     open_files_limit    = 65535
     back_log = 1024
-    max_connections = 512
+    max_connections = 1024
     max_connect_errors = 1000000
     table_open_cache = 1024
     table_definition_cache = 1024

#  take tag Push to remote server 
$ git push -u origin --tags
$ git push -u gitee --tags

#  Online server authentication （ Tulio ）

O & M management server update code , And switch to the specified tag( Be careful ！ Use Git Be sure to develop the habit of git pull This habit ).

##  Update code 
$ git pull

##  Switch to v0.1
$ git checkout -b v0.1

Through the above waves of operation , We can see , All our configuration changes have adopted Git management , The full lifecycle management of the configuration is completely recorded , By branching out to the warehouse or tag, It is convenient for us to switch to any recorded state .

High availability deployment MySQL( Reserved pit )

There is no high availability deployment requirement for the time being , Therefore, high availability mode is not involved MySQL Deployment of , But there are some thoughts left to occupy the pit .

The current practice

  • Don't make trouble for yourself , Those who directly buy cloud service providers with high availability demand RDS.
  • I really need to build it myself , stay K8s Deploy master-slave outside the cluster .

Possible future directions

  • K8s Upper MySQL Master-slave deployment
  • Operator
  • Helm

remaining problems

This part is also about operation and maintenance MySQL Essential skills , Some of the content I have no experience to share , Some content will be in << be based on KubeSphere Of K8s The road of production practice >> Series of documents .

  • MySQL Database backup
  • MySQL High availability deployment
  • MySQL Safety reinforcement
  • MySQL tuning

MySQL performance ( The benchmark ) test

The O & M must be responsible for its own O & M environment know the score ,MySQL Before going online, you must perform a performance test ( The benchmark ), It is helpful to understand the ideal state of our database server . This introduction is only superficial , Just tell you some basic knowledge , More details 、 For more in-depth content, please refer to other more professional documents .

performance ( The benchmark ) Test tool installation

Tool selection (sysbench)

  • Cloud vendors use this tool to show the performance of their own database products
  • It is said that many DBA Also like to use

sysbench Tool installation

  • Installation tools 
#  Import software source 
$ curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash

#  install sysbench
$ yum install sysbench -y
  • verification - Execute the command to view the version 
$ sysbench --version
sysbench 1.0.20

performance ( The benchmark ) test

Test plan

  • Test parameters

  • indicators value
    Number of threads 8/16/32
    Single table data volume 100000
    Number of tables 16

    Performance indicators

    indicators explain
    TPSTransactions Per Second , That is, the number of transactions executed by the database per second , With commit The number of successes shall prevail .
    QPSQueries Per Second , That is, the database executes every second SQL Count （ contain insert、select、update、delete etc. ）.
    RTResponse Time , response time . Including average response time 、 Minimum response time 、 Maximum response time 、 The percentage of queries per response time . What needs more attention is , front 95-99% The maximum response time of . Because it determines the short board in most cases .
    Concurrency Threads Concurrency , Number of query requests that can be processed per second .

Prepare test data

Use us in k8s Database created on , Involving database operation commands , need terminal Log in to the container and run .

Create a test database in advance sbtest, And give root From random IP Remote management of all databases .

Don't do this in the production environment , Be sure to follow the principle of minimization ！

# bash
[email protected]:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database sbtest;
Query OK, 1 row affected (0.02 sec)

mysql> grant all privileges on *.* to 'root'@'%' identified by '[email protected]' with grant option;
Query OK, 0 rows affected, 1 warning (0.02 sec)
  • Test whether the database can connect 
#  install mysql client , The following example is in k8s Installed on node , Since the system is minimal to install , All will install many dependencies . The actual test can play a role mysql Of pod Or something else mysql Client tools .

$ yum install mysql -y

#  test MySQL Service connectivity  -h  yes k8s Node IP -P  yes mysql The port number of the external service 

$ mysql -h 192.168.9.91 -P 32529 -u root -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>
  • Prepare test data 
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root [email protected] --mysql-db=sbtest --table-size=100000 --tables=16 --threads=8 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_common.lua prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Initializing worker threads...

Creating table 'sbtest6'...
Creating table 'sbtest2'...
Creating table 'sbtest8'...
Creating table 'sbtest3'...
Creating table 'sbtest7'...
Creating table 'sbtest5'...
Creating table 'sbtest1'...
Creating table 'sbtest4'...
Inserting 100000 records into 'sbtest3'
Inserting 100000 records into 'sbtest6'
Inserting 100000 records into 'sbtest1'
Inserting 100000 records into 'sbtest4'
Inserting 100000 records into 'sbtest7'
Inserting 100000 records into 'sbtest5'
Inserting 100000 records into 'sbtest2'
Inserting 100000 records into 'sbtest8'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest11'...
Inserting 100000 records into 'sbtest11'
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest1'...
Creating a secondary index on 'sbtest6'...
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest2'...
Creating a secondary index on 'sbtest8'...
Creating table 'sbtest13'...
Inserting 100000 records into 'sbtest13'
Creating table 'sbtest9'...
Inserting 100000 records into 'sbtest9'
Creating table 'sbtest14'...
Creating table 'sbtest12'...
Inserting 100000 records into 'sbtest14'
Inserting 100000 records into 'sbtest12'
Creating table 'sbtest15'...
Inserting 100000 records into 'sbtest15'
Creating table 'sbtest16'...
Creating table 'sbtest10'...
Inserting 100000 records into 'sbtest16'
Inserting 100000 records into 'sbtest10'
Creating a secondary index on 'sbtest11'...
Creating a secondary index on 'sbtest13'...
Creating a secondary index on 'sbtest9'...
Creating a secondary index on 'sbtest12'...
Creating a secondary index on 'sbtest14'...
Creating a secondary index on 'sbtest15'...
Creating a secondary index on 'sbtest10'...
Creating a secondary index on 'sbtest16'...
  • Perform the test -8 Thread test 
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root [email protected] --mysql-db=sbtest --table-size=100000 --tables=16 --threads=8 --events=999999999 --report-interval=10 --time=100  /usr/share/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 8
Report intermediate results every 10 second(s)
Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 8 tps: 88.46 qps: 1782.38 (r/w/o: 1249.19/355.46/177.73) lat (ms,95%): 267.41 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 8 tps: 84.31 qps: 1678.47 (r/w/o: 1173.42/336.43/168.62) lat (ms,95%): 277.21 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 8 tps: 70.20 qps: 1413.82 (r/w/o: 990.21/283.20/140.40) lat (ms,95%): 369.77 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 8 tps: 47.30 qps: 946.00 (r/w/o: 662.20/189.20/94.60) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 8 tps: 43.80 qps: 875.99 (r/w/o: 613.19/175.20/87.60) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 8 tps: 60.70 qps: 1213.08 (r/w/o: 849.69/242.00/121.40) lat (ms,95%): 411.96 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 8 tps: 53.90 qps: 1078.22 (r/w/o: 754.42/216.00/107.80) lat (ms,95%): 376.49 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 8 tps: 56.49 qps: 1127.98 (r/w/o: 790.11/224.88/112.99) lat (ms,95%): 397.39 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 8 tps: 50.60 qps: 1014.59 (r/w/o: 709.56/203.82/101.21) lat (ms,95%): 434.83 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 8 tps: 54.70 qps: 1093.12 (r/w/o: 765.22/218.50/109.40) lat (ms,95%): 390.30 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            85582
        write:                           24452
        other:                           12226
        total:                           122260
    transactions:                        6113   (61.10 per sec.)
    queries:                             122260 (1221.96 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          100.0494s
    total number of events:              6113

Latency (ms):
         min:                                   35.63
         avg:                                  130.89
         max:                                  951.86
         95th percentile:                      390.30
         sum:                               800129.59

Threads fairness:
    events (avg/stddev):           764.1250/4.14
    execution time (avg/stddev):   100.0162/0.01
  • Perform the test -16 Thread test 
    $ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root [email protected] --mysql-db=sbtest --table-size=100000 --tables=16 --threads=16 --events=999999999 --report-interval=10 --time=100  /usr/share/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options: Number of threads: 16 Report intermediate results every 10 second(s) Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 16 tps: 114.41 qps: 2310.22 (r/w/o: 1621.18/458.63/230.41) lat (ms,95%): 369.77 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 16 tps: 106.35 qps: 2111.86 (r/w/o: 1474.74/424.41/212.71) lat (ms,95%): 383.33 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 16 tps: 80.40 qps: 1612.01 (r/w/o: 1129.21/322.00/160.80) lat (ms,95%): 623.33 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 16 tps: 63.40 qps: 1266.80 (r/w/o: 886.80/253.20/126.80) lat (ms,95%): 539.71 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 16 tps: 57.20 qps: 1145.91 (r/w/o: 802.74/228.78/114.39) lat (ms,95%): 549.52 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 16 tps: 69.91 qps: 1408.31 (r/w/o: 987.57/280.92/139.81) lat (ms,95%): 511.33 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 16 tps: 78.00 qps: 1547.22 (r/w/o: 1080.51/310.70/156.00) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00 [ 80s ] thds: 16 tps: 79.50 qps: 1599.87 (r/w/o: 1122.58/318.29/159.00) lat (ms,95%): 520.62 err/s: 0.00 reconn/s: 0.00 [ 90s ] thds: 16 tps: 67.80 qps: 1354.83 (r/w/o: 947.62/271.61/135.60) lat (ms,95%): 539.71 err/s: 0.00 reconn/s: 0.00 [ 100s ] thds: 16 tps: 73.90 qps: 1474.10 (r/w/o: 1030.80/295.50/147.80) lat (ms,95%): 502.20 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 110950 write: 31700 other: 15850 total: 158500 transactions: 7925 (79.00 per sec.) queries: 158500 (1580.05 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.)

General statistics: total time: 100.3103s total number of events: 7925

Latency (ms): min: 41.24 avg: 202.44 max: 1198.81 95th percentile: 511.33 sum: 1604328.52

Threads fairness: events (avg/stddev): 495.3125/4.03 execution time (avg/stddev): 100.2705/0.03


-  Perform the test -32  Thread test 

```shell
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=[email protected] --mysql-db=sbtest --table-size=100000 --tables=16 --threads=32 --events=999999999 --report-interval=10 --time=100  /usr/share/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 32
Report intermediate results every 10 second(s)
Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 32 tps: 140.10 qps: 2825.04 (r/w/o: 1981.25/560.39/283.39) lat (ms,95%): 450.77 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 32 tps: 124.41 qps: 2515.49 (r/w/o: 1763.43/503.24/248.82) lat (ms,95%): 549.52 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 32 tps: 95.90 qps: 1887.10 (r/w/o: 1316.70/378.60/191.80) lat (ms,95%): 733.00 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 32 tps: 81.80 qps: 1656.59 (r/w/o: 1164.89/328.10/163.60) lat (ms,95%): 707.07 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 32 tps: 82.60 qps: 1638.41 (r/w/o: 1143.51/329.70/165.20) lat (ms,95%): 657.93 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 32 tps: 94.34 qps: 1905.84 (r/w/o: 1336.62/380.65/188.58) lat (ms,95%): 623.33 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 32 tps: 87.86 qps: 1739.86 (r/w/o: 1215.31/348.73/175.82) lat (ms,95%): 634.66 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 32 tps: 84.40 qps: 1705.48 (r/w/o: 1196.49/340.20/168.80) lat (ms,95%): 759.88 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 32 tps: 80.50 qps: 1580.71 (r/w/o: 1101.70/318.00/161.00) lat (ms,95%): 612.21 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 32 tps: 81.40 qps: 1661.90 (r/w/o: 1167.00/332.10/162.80) lat (ms,95%): 707.07 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            133924
        write:                           38264
        other:                           19132
        total:                           191320
    transactions:                        9566   (95.33 per sec.)
    queries:                             191320 (1906.56 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          100.3457s
    total number of events:              9566

Latency (ms):
         min:                                   51.94
         avg:                                  335.14
         max:                                 1405.78
         95th percentile:                      657.93
         sum:                              3205913.85

Threads fairness:
    events (avg/stddev):           298.9375/5.15
    execution time (avg/stddev):   100.1848/0.14

MySQL Vessel performance monitoring chart .

kubesphere-projects-lstack-statefulsets-mysql-74

Clean up the test data ( To ensure more accurate data , It is recommended to clean up the data before each test , Prepare the data , test ).

$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root [email protected] --mysql-db=sbtest --table-size=100000 --tables=16 --threads=32 --events=999999999 --report-interval=10 --time=100  /usr/share/sysbench/oltp_read_write.lua cleanup
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Dropping table 'sbtest1'...
Dropping table 'sbtest2'...
Dropping table 'sbtest3'...
Dropping table 'sbtest4'...
Dropping table 'sbtest5'...
Dropping table 'sbtest6'...
Dropping table 'sbtest7'...
Dropping table 'sbtest8'...
Dropping table 'sbtest9'...
Dropping table 'sbtest10'...
Dropping table 'sbtest11'...
Dropping table 'sbtest12'...
Dropping table 'sbtest13'...
Dropping table 'sbtest14'...
Dropping table 'sbtest15'...
Dropping table 'sbtest16'...

test result

Summary and comparison of results .

Number of pressure measuring threads TPSQPS Delay
8611221130
16791580202
32951906335

It is suggested that according to the test results , tuning ！

summary

This article introduces in detail Git Common operations 、 How to store and synchronize code in multiple online code repositories , It also introduces GitOps And demonstrates how to use GitOps The idea is original K8s Upper Department MySQL service . Last , Demonstrated MySQL Common performance testing tools sysbench Installation and basic use of .

My years of operation and maintenance experience and ideas run through the full text .

This article by the blog one article many sends the platform OpenWrite Release ！

原网站

版权声明
本文为[Kubesphere cloud native]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/161/202206100847205157.html

边栏推荐

猜你喜欢

随机推荐