Practice of curve replacing CEPH in Netease cloud music

Curve  Block storage has been used online in the production environment for nearly three years , It has withstood the test of various abnormal and extreme scenes , Performance and stability exceed core business requirements

Netease cloud music background

Netease cloud music is one of the leading online music platforms in China , Provide an interactive content community for music lovers . Netease cloud music has created a large 、 Energetic and strong 、 Fast growing business , Provide users with community-centered online music services and social entertainment services . Its iconic key products include  “ NetEase cloud music ”  And affiliated social entertainment products , Such as  “LOOK  live broadcast ”、“ acoustic wave ”  And  “ Yinjie ”, Let music lovers explore independently through technology driven tools 、 enjoy 、 Share and create different music and music derivative content , And interact with others .

Cloud music cloud disk business background

Cloud music services using cloud disk mainly include the main station 、UGC、 Qu Ku, etc  Java  application , The main station is the core business of cloud music , The highest level of  SLA  guarantee （ Annual availability  >=99.99%）, In the face of providing a stable cloud music experience with hundreds of millions of users , This has always been our key and difficult point .2019  Before, cloud music mainly used  Ceph  Cloud disk , as everyone knows ,Ceph  Performance defects exist in large-scale scenarios , And it's hard to guarantee that we are in all kinds of anomalies  ( Bad disk slow disk 、 Storage downtime 、 Storage network congestion, etc )  The cloud disk in the scene  IO  Response delay is not affected ;Ceph  Cloud disk  IO  Jitter problem , We have tried to spend a lot of manpower on Optimization and transformation , But they are only slightly relieved , It can't be solved completely ; Performance problems are also analyzed and optimized with a lot of manpower , But it still can not meet the expectations , That's why we set up a project to understand  Curve  Block storage distributed storage system .

Curve  Introduction to block storage

Curve  Block storage can be well adapted to mainstream cloud computing platforms , And it has high performance 、 Easy operation and maintenance 、 Stable without jitter . We are in practical application , Use  Curve  Block storage docking  Cinder  As the backend of virtual machine cloud disk storage , docking  Nova  As a virtual machine system disk , docking  Glance  As a mirror storage backend . In the process of creating virtual machine ,Nova  Will pass  Curve  Block storage provides  Python SDK  Clone a new volume for use as a virtual machine system disk . In the process of creating the cloud disk ,Cinder  Will pass  Python SDK  Create an empty volume or clone a new volume from an existing volume snapshot , After that, it can be attached to the virtual machine and used as a cloud disk . Virtual machine uses  Libvirt  As a virtualization control service , Use  QEMU/KVM  As a virtualization engine .Curve  The block is stored as  Libvirt/QEMU  Provides a driver library , After compilation, you can directly use  Curve  Volume as remote storage , There is no need to put  Curve  Block storage volumes are mounted locally .

Why choose  Curve

1.  Business side

i.  According to our cloud music application scenarios ,Ceph  There are two main cloud disks

Pain points

：

• Poor performance
  ： Due to the poor performance of a single volume （ Mainly  IO  High delay ,IOPS  Couldn't get on , And vulnerable to other high load volumes in the cluster ）, Therefore, it can only be used for the system disk , Or it can be used to print logs for cloud disk supply , Unable to support the use of middleware business .

• IO  shake
  ： After our observation, we found that  IO  Delay exceeds  2s  It may cause disk  util 100%, The business will alarm in a large area , Request to pile up , In severe cases, it will cause avalanche effect ; According to the former  2  Years of observation ,Ceph  Cloud disk  IO  Jitter is very frequent （ Basically every month ）, The duration of jitter also reaches the level of minutes , Therefore, many core applications have switched to local storage to avoid similar problems .

ii. Curve  Cloud disk

advantage

：

• shake
  ： Since using  Curve  After cloud disk , disk  IO util  Monitoring has never been caused by distributed storage systems  100%  The alarm , The stability of business operation has been greatly improved , The core business has gradually moved back to  Curve  Cloud disk （ After all, the high space utilization of cloud disk 、 reliability 、 Portability 、 Rapid recovery capability is also highly valued by the business ）.

• performance
  ： Under the same hardware ,Curve  Single volume performance is  Ceph  Roll up  2  times  +, The delay is also much lower than  Ceph, Refer to the following figure for specific performance comparison ：

2.  Operation and maintenance side

i.  According to our cloud music operation and maintenance scenario ,Ceph  Of

Pain points

Mainly as follows ：

• Service upgrade
  ： Common scenarios that require upgrading the client include  bug  Repair 、 New function enhancement and version upgrade , We met a  Ceph  Community message module  32  Bit sequence number overflow  bug, The  bug  It will appear on the long-running client , cause  IO hang, Both the client and the server need to be updated to solve the problem . There are two options for updating the client , First, restart the virtual machine  QEMU  process , Second, perform hot migration operations on virtual machines  live migration, These two operations are relatively feasible in a few virtual machine scenarios , But if you do this for hundreds of virtual machines , Obviously, the operability is very low , Business is clearly unacceptable . In addition, the server upgrade is restarting  OSD  Process will also cause certain  IO  shake , It is necessary to operate in the low peak period of business , And the business needs to shut down the disk temporarily  util  The alarm .

• performance
  ： The O & M personnel mainly focus on the overall performance of the storage cluster , If the total capacity of the cluster does not match the total performance , This can easily lead to insufficient performance when the capacity is sufficient , Or create fewer volumes, resulting in a waste of capacity , You can either continue to create volumes that will affect a single volume  IO  Delay and throughput , in addition  Ceph  When the number of cluster volumes reaches a certain scale , As the number of volumes increases , The overall performance of the cluster is also declining , This results in a greater impact on the performance of a single volume .

• Algorithm
  ： Limited by  CRUSH  Algorithmic limitations ,Ceph  Of  OSD  The data distribution is very uneven , Serious waste of space , According to our observation , The highest and lowest  OSD  The difference in space utilization can reach  50%, Data balancing is often required , However, a large number of data migration operations will occur in the process of data balancing , Lead to  IO  shake , In addition, data balance cannot be solved perfectly  OSD  Unbalanced capacity utilization .

• IO  shake
  ： Change the bad plate , The node is down , high  IO  load , Capacity expansion （ No new  pool, Too many new  pool  It can lead to  OpenStack  Maintenance becomes complicated ） Data equalization , Network card packet loss , Slow disk, etc .

ii.  relatively speaking  Curve  It has remarkable advantages in the above aspects

advantage

：

• Service upgrade
  ： The client supports hot upgrade , In operation  QEMU  The process does not need to be restarted , There is no need to migrate , The millisecond impact is almost insensitive to the business in the virtual machine , For architecture design related to thermal upgrade, please refer to ①.Curve  When upgrading the server , Thanks to the  quorum  Mechanism consistency protocol  raft, As long as you upgrade by replica domain , It can guarantee the business  IO  The effect is on the second level ,IO  The delay does not exceed  2s  It won't lead to  util 100%.

• performance
  ：Curve  The cluster can operate at the same capacity , Create more volumes , And maintain stable performance output .

• Algorithm ：Curve  Data distribution is centralized  MDS  The service is carried out , It can guarantee a very high balance , The highest and lowest  chunkserver  The deviation of space utilization rate shall not exceed  10%, There is no need to perform data balancing operation .

• IO  shake
  ：Ceph  Cloud disks are prone to  IO  Jittery scene ,Curve  Cloud disk performance is more stable ,Curve VS Ceph  The details are as follows: ：

Use  Curve  The result of landing

Curve  Block storage has been used online in the production environment for nearly three years , It has withstood the test of various abnormal and extreme scenes , Performance and stability exceed core business requirements , No obvious fault occurs under common fault scenarios  IO  shake , The upgrade of the server and client versions has not affected the normal operation of the business , This fully proves that our choice at that time was correct , Also thank you  Curve  The help given by the students of the team in the process of our use . at present ： Cloud music use  Curve  Block storage serves as the cloud disk and system disk of the virtual machine , The system disk is usually of fixed capacity  40GB  or  60GB  Two specifications , The cloud disk has the smallest capacity  50GB, The biggest support  4TB（ This is a soft limitation ,Curve  The cloud disk actually supports the creation of  PB  Level volume ）.

Follow up planning

combination  Curve  Block storage ：

• Exploration is based on  Curve  Cloud native middleware scenario of block storage , For example, the transformed  Redis、Kafka、 Services such as message queuing run in  Curve  Block storage on a volume , Reduce failover time .

• Online based on  CurveBS+PolarFS+MySQL  Cloud native database .

• Other stock uses  Ceph  Cloud disk 、 The local storage virtual machine is switched to  Curve  Block storage volume .

at present  Curve  The team is also fully developing the shared file storage service , Netease is based on  OpenStack  Private cloud  2.0  The platform has evolved to be based on  Kubernetes  Of  3.0  platform , Business for  ReadWriteMany  The type of  PVC  The demand for volumes has become more and more urgent ,Curve  Team developed  Curve  Distributed shared file system , The system supports storing data in  Curve  Block storage backend or compatible  S3  Protocol object storage service , It will also be launched as soon as possible .

Reference resources ：

① https://github.com/opencurve/curve/blob/master/docs/cn/nebd.md

 

• GitHub
  ：
  https://github.com/opencurve/curve

• Wechat group ：
  Please search add or search group assistant wechat  OpenCurve_bot