Grayscale upgrade tidb operator

If you want to upgrade TiDB Operator To the latest version , At the same time, I hope to control the impact of the upgrade , Avoid damage to the whole Kubernetes All in the cluster TiDB Clusters have unpredictable effects , It can be upgraded by grayscale upgrading TiDB Operator. After using grayscale upgrade , You can confirm in the grayscale deployment cluster TiDB Operator The impact of the upgrade , Upon confirmation of TiDB Operator After the new version works stably , Again Normal upgrade TiDB Operator.

TiDB Operator Currently, only some components can be grayed out , namely  tidb-controller-manager  and  tidb-scheduler, No support for Enhanced StatefulSet controller and Access controller Grayscale upgrade .

In the use of TiDB Operator when ,tidb-scheduler  It's not necessary to use . You can refer to  tidb-scheduler And default-scheduler, Confirm whether to deploy  tidb-scheduler.

The first 1 Step ： For the current TiDB Operator To configure selector And perform the upgrade

At present TiDB Operator Of  values.yaml  in , Add the following selector To configure ：

controllerManager: selector: - version!=canary

Reference resources Online upgrade or Offline upgrade , For the current TiDB Operator Perform the upgrade steps ：

helm upgrade tidb-operator pingcap/tidb-operator --version=${chart_version} -f ${HOME}/tidb-operator/values-tidb-operator.yaml

The first 2 Step ： Deploy grayscale TiDB Operator

1. Reference resources Online deployment TiDB Operator  Of the 1 And the first step 2 Step , Get the grayscale version you want to deploy TiDB Operator Of  values.yaml  file , And in  values.yaml  Add the following configuration .

   controllerManager: selector: - version=canary appendReleaseSuffix: true #scheduler: # create: false # If you don't need to `tidb-scheduler`, Set this value to false advancedStatefulset: create: false admissionWebhook: create: false

   appendReleaseSuffix  I need to set to  true.

   If you don't need grayscale upgrade tidb-scheduler, You can set  scheduler.create: false. If you need a grayscale upgrade tidb-scheduler, To configure  scheduler.create: true, Will create a file named  { { .scheduler.schedulerName }}-{ {.Release.Name}}  Of scheduler. If you want to deploy in grayscale TiDB Operator Use this scheduler, Need configuration TidbCluster CR Medium  spec.schedulerName  For this scheduler Name .

   Because grayscale upgrade does not support enhanced StatefulSet Controller and access controller , You have to configure  advancedStatefulset.create: false  and  admissionWebhook.create: false.

   For more information about the parameters related to gray-scale deployment , May refer to Use multiple sets TiDB Operator Manage different TiDB colony - Related parameters .

2. stay Different namespace  in （ for example  tidb-admin-canary）, Use Different  Helm Release Name（ for example  helm install tidb-operator-canary ...） Deploy grayscale TiDB Operator：

   helm install tidb-operator-canary pingcap/tidb-operator --namespace=tidb-admin-canary --version=${operator_version} -f ${HOME}/tidb-operator/${operator_version}/values-tidb-operator.yaml

   take  ${operator_version}  Replace with the one you need to upgrade to TiDB Operator Version number .

The first 3 Step ： Test grayscale TiDB Operator ( Optional )

Upgrade normally TiDB Operator front , Gray scale deployment can be tested TiDB Operator Whether it works stably . The components that support testing are tidb-controller-manager and tidb-scheduler.

1. If you need to test grayscale deployment tidb-controller-manager, This can be done by the following command , For a TiDB Cluster settings label：

   kubectl -n ${namespace} label tc ${cluster_name} version=canary

   By viewing the two deployed tidb-controller-manager Log , You can confirm this setting label Of TiDB Clusters have been deployed by grayscale TiDB Operator management . The steps to view the log are as follows ：

   1. View the current TiDB Operator Of tidb-controller-manager Log :

      kubectl -n tidb-admin logs tidb-controller-manager-55b887bdc9-lzdwv

      The expected output is as follows ：

      I0305 07:52:04.558973 1 tidb_cluster_controller.go:148] TidbCluster has been deleted tidb-cluster-1/basic1

   2. View grayscale deployment TiDB Operator Of tidb-controller-manager Log :

      kubectl -n tidb-admin-canary logs tidb-controller-manager-canary-6dcb9bdd95-qf4qr

      The expected output is as follows ：

      I0113 03:38:43.859387 1 tidbcluster_control.go:69] TidbCluster: [tidb-cluster-1/basic1] updated successfully

2. If you need to test grayscale deployment tidb-scheduler, This can be done by the following command , For a TiDB Cluster modification  spec.schedulerName  by  tidb-scheduler-canary：

   kubectl -n ${namespace} edit tc ${cluster_name}

   After modification , Components in the cluster will be upgraded in a rolling manner , By viewing the grayscale deployment TiDB Operator Of  tidb-scheduler  Log , You can confirm that the cluster has used grayscale  tidb-scheduler：

   kubectl -n tidb-admin-canary logs tidb-scheduler-canary-7f7b6c7c6-j5p2j -c tidb-scheduler

3. After the test is completed , You can undo the changes in the previous two steps , Reuse the current TiDB Operator To manage TiDB colony .

   kubectl -n ${namespace} label tc ${cluster_name} version-

   kubectl -n ${namespace} edit tc ${cluster_name}

The first 4 Step ： Normal upgrade TiDB Operator

Confirm the grayscale deployment TiDB Operator After normal operation , It can be upgraded normally TiDB Operator.

1. Delete grayscale deployment TiDB Operator：

   helm -n tidb-admin-canary uninstall ${release_name}

2. normal upgrade TiDB Operator.