Kubernetes' simplified data storage storageclass (creation, deletion and initial use)

kubernetes Simplified data storage StorageClass（ Create and delete as well as initial use ）

List of articles

• Preface
• One 、StorageClass What is it? ？
• Two 、 Use steps
  • 1. Import and stock in
  • 2. Read in the data
  • 3
  • 4
  • 5
• summary

---

Preface

k8s There are many new concepts , And these concepts make us learn k8s More costs are needed , This paper is mainly about k8s Some new concepts involved in data persistent storage ：pv,pvc,sc,svc Discuss , Thus in the actual production activities , Better use k8s colony .

---

One 、StorageClass What is it? ？

Say first conclusion ,StorageClass Provides a description store for Administrators " class " Methods , Realize the dynamic supply of storage , Simply speaking ,StorageClass Can be based on pvc To automatically create pv, Ease the creation of cluster administrators pv The burden of . Simply say human words , Is managing k8s In clusters , No need to create pv 了 , Just create one pvc That's all right. .

that , This leads to two new concepts , What is? pv？ What is? pvc？

（1）pv

pv=persistentVolume, Is an abbreviation , Chinese meaning persistent storage ,PV It's an abstraction of the underlying network shared storage , Define shared storage as “ resources ”, such as Node It's also a resource that container applications can consume .PV Created and configured by the Administrator , It is directly related to the implementation of shared storage .

kubernetes Supported by PV The types are as follows ：

◎ AWSElasticBlockStore：AWS Provided by the public cloud ElasticBlockStore.

◎ AzureFile：Azure Provided by the public cloud File.

◎ AzureDisk：Azure Provided by the public cloud Disk.

◎ CephFS： An open source shared storage system .

◎ FC（Fibre Channel）： Optical storage devices .

◎ FlexVolume： A plug-in storage mechanism .

◎ Flocker： An open source shared storage system .

◎ GCEPersistentDisk：GCE Provided by the public cloud PersistentDisk.

◎ Glusterfs： An open source shared storage system .

◎ HostPath： Host directory , For stand-alone testing only .

◎ iSCSI：iSCSI The storage device .

◎ Local： Local storage devices , Currently, you can specify the block （Block） Equipment supply Local PV, Or through community development sig-storage-local-static-provisioner plug-in unit （ https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner ） To manage Local PV Life cycle of .

◎ NFS： Network file system .

◎ Portworx Volumes：Portworx Storage services provided .

◎ Quobyte Volumes：Quobyte Storage services provided .

◎ RBD（Ceph Block Device）：Ceph Block storage .

◎ ScaleIO Volumes：DellEMC Of storage devices .

◎ StorageOS：StorageOS Storage services provided .

◎ VsphereVolume：VMWare Storage system provided .

As you can see above ,nfs,iscsi,ceph These commonly used network storage or block storage will be regarded as a resource , It means being k8s Think of it as an available resource .

A simple pv create a file , The contents are as follows ：

[r[email protected] mysql]# cat pv_mysql.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: nfs-pv-test
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce      
  capacity:
    storage: 1.5Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /data/nfs_sc/nfs-pv-test
    server: 192.168.217.16

Above this pv create a file , There are many parameters , The parameters that need attention are ：

1, capacity Its value is pod How much data is allowed to persist , If the pv By some pvc call , that , Will only be used 1.5G Space .

2, Storage volume mode （Volume Mode）

volumeMode=xx, Options include Filesystem（ file system ） and Block（ Block device ）, The default value is FileSystem. The above file is used for network storage nfs, therefore , Omit here , Default for use filesystem

3,

Access pattern （Access Modes）

Used to describe the application's access to storage resources .

◎ ReadWriteOnce（RWO）： read-write permission , And can only be single Node mount .

◎ ReadOnlyMany（ROX）： Read only permission , Allow to be more than one Node mount .

◎ ReadWriteMany（RWX）： read-write permission , Allow to be more than one Node mount .

The above example uses readwriteonce , This one is also commonly used , Another common one is readwritemany, but ,many It may cause confusion in later management , therefore ,once Is the most used , If you don't want to make trouble for yourself ,once That's enough. , Do not use many.

4、

Storage class （Class）

Set the category of storage , adopt storageClassName The parameter is assigned to a StorageClass The name of the resource object , Having a specific class of PV Can only be associated with PVC Binding . Unbound class PV Only with no request for any kind of PVC Binding .

I've already said that , If there is a setting StorageClass, And this StorageClass If it is set as the default , This can not be specified . If there is no default StorageClass, that , You must specify .

How to choose StorageClass？

[[email protected] mysql]# k get sc -A
NAME              PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
mynfs             mynfs                                 Delete          Immediate           true                   4h17m
nfs (default)     nfs                                   Delete          Immediate           true                   4h19m
nfs-provisioner   choerodon.io/nfs-client-provisioner   Delete          Immediate           false                  3d20h
nfs-sc            storage.pri/nfs                       Delete          Immediate           true                   5h5m

For example, my test system , I installed many StorageClass, Example pv The file uses the default nfs（default） 

5、

Recovery strategy （Reclaim Policy）

adopt persistentVolumeReclaimPolicy Field settings ,

◎ Retain Retain ： Keep the data , It needs to be handled by hand .

◎ Recycle Recycle space ： Simple operation to clear files （ For example, to perform rm -rf /thevolume/* command ）.

◎ Delete Delete ： And PV Connected back-end storage complete Volume Delete operation of

EBS、GCE PD、Azure Disk、OpenStack Cinder Wait inside the equipment Volume clear ）.

The recycling strategy is set according to the actual production activities , If persistent data is important , that ,retain It's a good choice , This means , If it is retain Even if you delete pv, The data remains . This example uses retain

How to query pv Which strategy is used ？（ You can't always look at the example installation file above , What if I can't find it ？）

[[email protected] mysql]# k get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                          STORAGECLASS      REASON   AGE
mysql-pv-test                              1Gi        RWO            Retain           Terminating   database/mysql-pvc-test                        nfs-provisioner            34h
nfs-pv-test                                1536Mi     RWO            Retain           Terminating   database/nfs-pvc-test                          nfs                        3h56m
pvc-04203f8a-5907-48ce-9fc2-013e94313c3c   8Gi        RWO            Delete           Bound         kube-system/redis-data-redis-test-replicas-1   nfs-provisioner            3d11h
pvc-751a32b6-8706-477b-8cad-d71e8e9f3ab8   256Mi      RWO            Delete           Bound         kube-system/redis                              nfs-provisioner            3d11h
pvc-d5ea7d10-2ffa-402e-b3f1-8573a195ad6f   8Gi        RWO            Delete           Bound         kube-system/redis-data-redis-test-replicas-0   nfs-provisioner            3d11h
pvc-e1693689-b01b-4855-ab1c-b8f843be4e2e   8Gi        RWO            Delete           Bound         kube-system/redis-data-redis-test-replicas-2   nfs-provisioner            3d11h
pvc-f9193155-776c-42f4-a3f5-71e75f16416f   8Gi        RWO            Delete           Bound         kube-system/redis-data-redis-test-master-0     nfs-provisioner            3d11

that , I created nfs-pv-test This pv It is used retain Strategy .

How to change the strategy ？  again kubectl  apply -f Sample files are ok  , For example, modify as follows ：

[[email protected] mysql]# cat pv_mysql.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: nfs-pv-test1
spec:
  accessModes:
    - ReadWriteOnce      
  capacity:
    storage: 1.5Gi
  persistentVolumeReclaimPolicy: Recycle 
  storageClassName: nfs
  nfs:
    path: /data/nfs_sc/nfs-pv-test1
    server: 192.168.217.16

 

[[email protected] mysql]# k apply -f pv_mysql.yaml
The PersistentVolume "nfs-pv-test1" is invalid: spec.persistentVolumeReclaimPolicy: Unsupported value: "recycle": supported values: "Delete", "Recycle", "Retain"
[[email protected] mysql]# vim pv_mysql.yaml
[[email protected] mysql]# k apply -f pv_mysql.yaml
persistentvolume/nfs-pv-test1 created

The PersistentVolume "nfs-pv-test1" is invalid: spec.persistentVolumeReclaimPolicy: Unsupported value: "recycle": supported values: "Delete", "Recycle", "Retain"    This paragraph represents three strategies , Policy must specify , There is no default .

[[email protected] mysql]# k get pv -A
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                          STORAGECLASS      REASON   AGE
mysql-pv-test                              1Gi        RWO            Retain           Terminating   database/mysql-pvc-test                        nfs-provisioner            35h
nfs-pv-test                                1536Mi     RWO            Retain           Terminating   database/nfs-pvc-test                          nfs                        4h2m
nfs-pv-test1                               1536Mi     RWO            Recycle          Available                                                    nfs                        15s
pvc-04203f8a-5907-48ce-9fc2-013e94313c3c   8Gi        RWO            Delete           Bound         kube-system/redis-data-redis-test-replicas-1   nfs-provisioner            3d11h
pvc-751a32b6-8706-477b-8cad-d71e8e9f3ab8   256Mi      RWO            Delete           Bound         kube-system/redis                              nfs-provisioner            3d11h

  Here's a little bit of attention ,pv The name is nfs-pv-test1, The recycling strategy is Recycle, You need to pay attention to case . It and others pv Different states , Others are either bound, Or Terminating , And this is available, Because of this pv Not yet bound to any pvc. and Terminating Express this pv The mission is over , because , I used the default StorageClass,pod It's up ,pv And that's the end of your life cycle （ Did not continue bound The meaning of binding ）.

in addition , If resource provisioning uses dynamic mode , That is, the administrator has not defined in advance PV, Only through StorageClass Give it to the system to finish automatically PV Dynamic creation of , that PVC Then set Selector when , The system will not be able to supply any storage resources .

With dynamic supply mode enabled , Once the user has deleted PVC, Bound to it PV It will also be based on its default recycling strategy “Delete” Be deleted . If you need to keep PV（ User data ）, After the dynamic binding is successful , The user needs to generate the system automatically PV The recycling strategy of “Delete” Change to “Retain”. Talk is talk , If used default Of StorageClass, that , The strategy must be retain.

PV All stages of the life cycle

◎ Available： Available status , Not yet with one PVC binding .

◎ Bound： With a certain PVC binding .

◎ Released： The binding of PVC Has deleted , Resources have been released , But not recycled by the cluster .

◎ Failed： Automatic resource recovery failed

◎Terminating：pv It's over

 

In general ,bound It means normal , If used default Of StorageClass, that ,pv Because it is StorageClass Automatic start stop management , therefore ,Terminating It also means normal .

6、

Mount parameters （Mount Options）

Will be PV Mount to a Node Upper time , According to the characteristics of back-end storage , You may need to set additional mount parameters , According to the PV In the definition mountOptions Field to set . This example does not use .

---

 

（2）pvc

pvc=PersistentVolumeClaim, It is also an abbreviation , List of persistent storage requirements

PVC It is a user's request for storage resources “ apply ”, It's like Pod consumption Node Resources are the same ,PVC Be able to consume PV resources .PVC You can apply for specific storage and access patterns .

Example ：

[[email protected] mysql]# cat pvc_mysql.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc-test
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1.5Gi
  storageClassName: nfs

kubectl apply -f  pvc_mysql.yaml You can create this name nfs-pvc-test Of pvc.

[[email protected] mysql]# k get pvc -A
NAMESPACE     NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
database      mysql-pvc-test                     Bound    mysql-pv-test                              1Gi        RWO            nfs-provisioner   35h
database      nfs-pvc-test                       Bound    nfs-pv-test                                1536Mi     RWO            nfs               4h35m

You can see ,pvc It is related to namespace , Also automatically bound nfs-pv-test This pv 了 . The last side also suggests the use of StorageClass  , The name is nfs.

Key configuration

1、 Resource request （Resources）

Describe the request for storage resources , Currently only supported request.storage Set up , It's the size of the storage space , This example is not configured

2、 Access pattern （AccessModes）

Used to describe access to storage resources , And PV The settings are the same , This example is still configured with once

3、 Storage volume mode （Volume Modes）

Used to describe the PV Storage volume mode , Including file systems and block devices . This example uses the default , Still filesystem

4、PV Choose the conditions （Selector）

Through to Label Selector Set up , Can make PVC For all kinds of things that already exist in the system PV Screening .

You can use matchLabels and matchExpressions Set it up , If both fields are set , be Selector The logic is that two sets of conditions can be met at the same time to complete the matching

This example is not configured

5、 Storage class （Class）

PVC When defining, you can set the category of back-end storage you need （ adopt storageClassName Field assignment ）, To reduce the dependency on the details of back-end storage features . Only set up the Class Of PV To be selected by the system , And with the PVC Binding

PVC It can also be set without Class demand . If storageClassName The value of the field is set to null （storageClassName=""）, It means that PVC Do not require specific Class, The system will only select unset Class Of PV Match and bind with .PVC It can also be set completely without storageClassName Field , This will depend on whether the system is enabled with the name DefaultStorageClass Of admission controller Operate accordingly

This example configures StorageClass, And it USES the default.

6、 not enabled DefaultStorageClass

Equivalent to PVC Set up storageClassName The value of is empty （storageClassName=""）, That is to say, you can only choose not to set Class Of PV Match and bind with .

In this example, it can be specified as non default, such as nfs-sc

[[email protected] nfs-sc]# k get sc 
NAME              PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
mynfs             mynfs                                 Delete          Immediate           true                   4h57m
nfs (default)     nfs                                   Delete          Immediate           true                   4h59m
nfs-provisioner   choerodon.io/nfs-client-provisioner   Delete          Immediate           false                  3d21h
nfs-sc            storage.pri/nfs                       Delete          Immediate           true                   5h45m

Two 、 When you need to use StorageClass

1. Key configuration

StorageClass As an abstract definition of storage resources , Set for users PVC Apply to block the details of back-end storage , On the one hand, it reduces the user's attention to the details of storage resources , On the other hand, it reduces the manual management of administrators PV The job of , It's done automatically by the system PV Create and bind , Dynamic resource supply is realized .

StorageClass The definition mainly includes the name 、 The provider of back-end storage （privisioner） And back-end storage related parameter configuration .StorageClass Once created , You can't modify , If you need to modify , Can only delete rebuild . for example , Create a whole new StorageClass：

[[email protected] nfs-sc]# cat storageclass-nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mynfs
provisioner: mynfs
reclaimPolicy: Delete
allowVolumeExpansion: True  # allow pvc Capacity expansion after creation 

Key configuration

1、 Provider （Privisioner）

Describe the provider of the storage resource , It can also be seen as a back-end storage driver .

2、 Parameters （Parameters）

Parameter settings of the backend storage resource provider , Different Provisioner Including different parameter settings . Some parameters can be set without display ,Provisioner Its default value will be used . This example uses the default configuration , No parameters are used . Generally, there is no need to set parameters .

3,allowVolumeExpansion: True  # allow pvc Capacity expansion after creation

If you are not sure whether the storage space is enough （ such as ,nfs）, Please set it to true.

4,reclaimPolicy: Delete Here we still look at our own needs ,delete Basically enough , If you care about the data , then retain.

 

2. Set default StorageClass

for example , I have 4 individual StorageClass, adopt k get sc -A The order found out ：

[[email protected] nfs-sc]# k get sc -A
NAME              PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
mynfs             mynfs                                 Delete          Immediate           true                   5h13m
nfs (default)     nfs                                   Delete          Immediate           true                   5h15m
nfs-provisioner   choerodon.io/nfs-client-provisioner   Delete          Immediate           false                  3d21h
nfs-sc            storage.pri/nfs                       Delete          Immediate           true                   6h

that , I want to set nfs-sc This is the default default, How to set it ？

kubectl patch storageclass nfs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Query again ：

[[email protected] nfs-sc]# k get sc -A
NAME               PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
mynfs              mynfs                                 Delete          Immediate           true                   5h16m
nfs (default)      nfs                                   Delete          Immediate           true                   5h17m
nfs-provisioner    choerodon.io/nfs-client-provisioner   Delete          Immediate           false                  3d21h
nfs-sc (default)   storage.pri/nfs                       Delete          Immediate           true                   6h3m

  That's not good , Two default, that , There will definitely be various problems in the future （k8s I don't know which one to use ）, hold nfs（default） Get rid of default, Simple ：

kubectl patch storageclass nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Query again , You have achieved your goal ：

[[email protected] nfs-sc]# k get sc -A
NAME               PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
mynfs              mynfs                                 Delete          Immediate           true                   5h18m
nfs                nfs                                   Delete          Immediate           true                   5h20m
nfs-provisioner    choerodon.io/nfs-client-provisioner   Delete          Immediate           false                  3d21h
nfs-sc (default)   storage.pri/nfs                       Delete          Immediate           true                   6h6m

 

 

---

summary

pv,pvc and StorageClass The relationship between the three is relatively close , But there are still some running rules that need to be highlighted ：

（1） The supply of resources

k8s Support two supply models of resources ： Static mode （Static） And dynamic mode （Dynamic）. The result of resource provision is to create good PV.

Static mode ： Cluster administrators create many by hand PV, In defining PV You need to set the characteristics of back-end storage .

Dynamic mode ： Cluster administrators do not need to create PV, But through StorageClass The setting of describes the back-end storage , To mark with a certain type . Demand at this time PVC Declare the type of storage , The system will automatically complete PV The creation of and PVC The binding of .PVC It can be stated that Class by "", Explain the PVC Do not use dynamic mode .

（2） Resource binding

In defining PVC after , The system will be based on PVC Requirements for storage of resources （ Storage and access patterns ） In the existing PV Choose one to satisfy PVC Required PV, Once found , It's time to PV Definition and PVC Binding , Applications can use this PVC 了 . If the system doesn't have this PV, be PVC We'll deal with it all the time Pending state , Until there is a qualified PV.PV Once bound to PVC On , Will be PVC Monopoly , Can't be compared with other PVC Binding . When PVC The storage space ratio of applications PV Less time , Whole PV The space can be PVC used , May cause waste of resources . If resource provisioning uses dynamic mode , Then the system is PVC Find the right StorageClass after , Will automatically create a PV And complete with PVC The binding of .

（3） Use of resources

Pod Use Volume Definition , take PVC Mount to a path in the container for use .Volume The type of Persistent VolumeClaim, There is a... Mounted in the container PVC after , Can be used exclusively and continuously . Multiple Pod Can be mounted to the same PVC On .

volumes:
  - name: pv
    persistentVolumeClaim:
      claimName: pvc

（4） Release resources

When the storage resources are used up , You can delete PVC, With this PVC The binding of PV Will be marked as “ Released ”, But not immediately with other PVC Binding . Before passing PVC The data written may still be retained on the storage device , Only after removal should PV To be used again .

（5） Recycling

about PV, The administrator can set the recycling policy , Used to set the PVC How to deal with legacy data after releasing resources . Only PV The storage space of is recovered , To supply a new PVC Bind and use .

In the static resource supply mode , adopt PV and PVC Complete the binding , And provide Pod The storage management mechanism used

In the dynamic resource supply mode , adopt StorageClass and PVC Complete resource dynamic binding （ Automatic system generation PV）, And provide Pod The storage management mechanism used .

（6）

If , Enable DefaultStorageClass The Cluster Administrator is required to define the default StorageClass. If there is no default in the system StorageClass, Is equivalent to not enabling DefaultStorageClass The situation of . If there is a default StorageClass, Then the system will automatically be PVC Create a PV（ Use the default StorageClass Back end storage ）, And bind them . The cluster administrator sets the default StorageClass See the above point in the summary for the method of , If the administrator will have more than one StorageClass All defined as default, Because it's not unique , The system will not be able to PVC Create the corresponding PV.

（7） not enabled DefaultStorageClass

Equivalent to PVC Set up storageClassName The value of is empty （storageClassName=""）, That is to say, you can only choose not to set Class Of PV Match and bind with .

---

---

These rules are very important , Also more tongue twister , It needs careful practice , Thus to k8s There is a correct understanding .