当前位置：网站首页>Topolvm: kubernetes local persistence scheme based on LVM, capacity aware, dynamically create PV, and easily use local disk

Topolvm: kubernetes local persistence scheme based on LVM, capacity aware, dynamically create PV, and easily use local disk

2022-06-27 01:00:00 【InfoQ】

R & D test scenario , The general pursuit is to quickly start the environment with one click , Horizontal dynamic replication , One person, one set , Use as you go , Go as soon as you use it . As the user , It doesn't care what the actual physical resources are , Where does the environment start , Just declare your own usage requirements . But as a solution builder and infrastructure brace , But we know , To provide a better solution , There are still many problems , And it's not easy .

For example, in the past , I was once puzzled by how to gracefully release the local physical disk for business use , Especially local HDD Data disk .

There is a background , our Kubernetes The R & D test cluster is built with the over guaranteed machines that have retired online , Then qiniu started cloud storage , So many of our machines are multi disk storage intensive machines ( Like hanging 12 block 4T Disk ). So how to make better use of these disks , It's a question .

One solution is to make these disks into network storage , And then through qiniu's own cloud service or ceph Wait for the system to provide , It's certainly possible . However, some businesses just want to use physical disks , So what should we do ？

throughout Kubernetes At present, several native solutions are provided , The author found that they are not perfect :

Emptydir
  Non persistent scheme ,Pod Delete ,EmptyDir It will be emptied . in addition Emptydir It uses rootfs Storage space , This space may be on the system disk , Therefore, it should be used with caution .

HostPath
  Persistence scheme , But the security risks are high , The government does not recommend . in addition , It is also inconvenient from the point of view of use . For example, as a user , You must first know the situation of the target system , Only after you know the specific drive letter and directory location can you HostPath Quote correctly in . But the reality is from the perspective of cluster security , The user may not have the right to log in to the machine directly . And even if your Pod Occupied the directory of a host computer , Nor can we rule out the second occupation by others , Or misoperation . therefore HostPath The usage scenarios for are actually very limited .

Local Persistent Volume
  Persistence scheme , With PVC/PV To use local storage . Good idea. , But it cannot be created dynamically PV, about PV For providers , The mental burden is high . Of course, the community now offers Local Static Provisioner, To some extent, it simplifies the mental burden of this piece , But you still need to plan directories or partitions in advance , It's not enough .

I think , Ideal local disk usage , Should be

Apply on demand , Spatial isolation , And automated lifecycle management

. In this way, it is convenient for the end user to use , It can also reduce operation and maintenance support , Increase of efficiency . There are three key technical points here :

Apply on demand
  It means that it is best to PVC+StorageClass To provide services , Achieve PV Dynamically create . And to achieve that ,
Capacity aware
  Is the key , Because it is obviously unreasonable to schedule to nodes with insufficient space . It's better to combine PVC Capacity requested +Node Remaining capacity on , Comprehensively select the best node to bind .

Spatial isolation
  Make sure that the space requested by the user is sufficient , Free from encroachment . From here , Simply use the directory of the file system as PV But it is obviously inappropriate not to isolate each other in capacity .

Automated lifecycle management
  dynamic Provisioning Is a strong demand .

Synthesize the above three points , We will find that based on  

LVM Or partition technology +CSI

  The implementation of the , When it is more in line with the above user experience , and

TopoLVM

It's such a project .

Address : https://github.com/topolvm/topolvm

TopoLVM Is based on LVM Of Kubernetes Localized disk solutions , With CSI The form is provided to users for use . Currently, the following functions are supported :

dynamic Provisioning

Support native block volumes （Raw Block Volume）

Volume Telescopic

The overall structure is as follows : 

It is worth noting that , In earlier versions , In order to dynamically perceive the remaining storage capacity on the node ,TopoLVM Designed a custom extension scheduler （ Upper figure topolvm-scheduler part ）, Convenient in Scheduling Stage is Pod Bind appropriate Node. And in the Kubernetes 1.21 after ,Kubernete Native support already  

Storage Capacity Tracking

The ability of , The whole implementation of this piece becomes much more elegant ,topolvm-Scheduler It's no longer necessary .

Of course , To realize TopoLVM The core principle of , Besides understanding CSI Write out of specification , The most important thing is to understand LVM Related technology . And it is precisely because through LVM Can dynamically create LV, Dynamic expansion and contraction ,TopoLVM To support dynamic Provisioning Related capabilities .

however , Although as an open source project TopoLVM Basically enough , But the richness is slightly insufficient . Recently, Boyun has also opened its cloud native local disk management solution Carina, It looks more perfect .

Project address : https://github.com/carina-io/carina

Carina In addition to providing based on LVM Out of the plan , It also supports raw disk partition , as well as IOPS Limit etc. , More features . The code organization specification is also more suitable for the cloud native community , The whole is also worth exploring .

Reference link