How to build a cloud storage cluster based on swift open source technology

With cloud computing 、 mobile communication 、IoT The development of , Traditional block device and file system access are facing more and more limitations , Object storage comes into being . Object storage enables applications or end devices to pass directly through web or http Access to data becomes possible . secondly , Due to the distributed storage characteristics of object storage , It is naturally suitable for the application scenario of large-scale unstructured data storage , Backup 、 file 、 File sharing, etc .

This article describes how to open source based Swift Object storage technology designs and builds a set of mass storage systems 、 Security 、 Highly reliable 、 Low cost cloud storage cluster .

Before building

Before formal design and build , Be sure to know the usage scenarios of the storage system and the storage requirements .

At present, cloud storage is mainly applicable to the following scenarios ：

• Provide services such as cloud disk and file sharing for internal use ;
• Backup and archive
• Provide cloud storage services as a public cloud manufacturer
• As part of the private cloud , Provide storage

For specific scenarios , We can consider specific requirements from the following two aspects ：

Storage requirements ：

• Capacity ： storage capacity 、 Expected future growth rate
• Data reliability requirements ： Replica Count 、Erasure Code

access ：

• Number of access users 、 Frequency of user access 、 Number of simultaneous users
• read 、 Write 、 modify
• Performance requirements for access ： Time delay 、 bandwidth
• Mainly large files , Or small files
• Availability requirements for access

The system design

Design storage system , In essence, it is in performance 、 Capacity 、 reliability 、 Usability 、 cost 、 Safety, etc trade-off, So as to design a set of systems that are most suitable for organizational needs . Don't expect one-size-fit-all. Before formal system implementation , A set of PoC System , The PoC The size ratio of the system to the actual system should be within 1：10 To 1：100 Between .

- Yes PoC The system can be tuned or even adjusted , In order to achieve the performance you require

- Extend the system , And continuous performance tuning , To keep your PoC Performance achieved when .

PoC After the system reaches the desired goal and operates stably , Right again PoC System expansion , Become an actual production system .

1） Server selection

A typical Swift In a storage cluster , Contains at least two types of nodes ： Storage nodes and proxy nodes . When selecting a server, you must consider the hard disk 、CPU、 Memory and network card . When we choose a storage node, we often first consider the hard disk , Then combine CPU、 Memory requirements , To determine the type and number of servers .

Hard disk

The required disk space can be calculated by the following formula ：

Disk space required （GB） = User's effective storage capacity （GiB）* replications * 1.087

among ,1.087 Consider the file system （XFS） Loss and GiB To GB Transformation .

Number of hard disks = Required hard disk space / Selected single hard disk capacity

Number of servers = Number of hard disks / Number of server disks

Note that the actual number of servers required is more than the servers calculated at this time , Mainly due to ：

1） The above calculation formula does not consider the number of hard disks occupied by the operating system .

2） In the above formula, only the usage space of the object is calculated .Swift Also need to consider Account and Container. Based on experience ,Account and Container The space used is about... Of the space used by the object 1%～2%. in addition , in consideration of Account and Container Store metadata , Need to visit frequently , So consider performance , In general use SSD To store Account and Container.

3） When determining the number of servers and the size of hard disks , You also need to consider the cost of a single hard disk 、 The cost of the server and the availability of the system .

CPU

CPU Check the number = （ Number of server hard disks * CPU The ratio of the number of cores to the number of hard disks ） / CPU The frequency of the nucleus

CPU The choice of the ratio of the number of cores to the number of hard disks depends on CPU Frequency and application scenarios ： Small files 、 High throughput 、 High concurrency scenarios require a higher proportion , Such as 2/3; Large files with low concurrency , You may only need 1/4. Based on experience , In general, choose 1/2 To 2/3 Between , Then optimize according to the actual situation .

Memory

The memory in the storage node is mainly used to run Swift Service process and cache XFS Of inode and swift Some directories of . Every inode The node size is 256 byte , Plus hash_dir, suffix_dir and partition Catalog , So each object needs about 1KB The cache of . The number of objects can be calculated according to the storage capacity and object size , To estimate the amount of memory required .

however , Considering the cost , Generally, the index data of all objects will not be cached . Based on experience , Each hard disk 1GB Memory should be a good starting point , We can do more optimization on this basis .

network card

According to the deployment mode and scale , In general, you can put Swift The cluster is divided into 3 To 5 A network ：

public The Internet ： External provision API visit

Internal network ： Used for data communication between nodes , In general , Traffic on the network is not encrypted

Manage the network ： Manage and monitor the status and services of the cluster .

Out of band network ：IPMI Manage the network

Data replication network ： On a large scale , A dedicated network may be considered to be responsible for data replication .

Based on the above network , Generally, special network card or VLAN.

As for proxy nodes , Because the proxy node handles the user's request , And write user data to or read data from the storage node , So right. CPU There's a huge demand for . meanwhile , Do not hard drive data IO operation , Therefore, the hard disk only needs to meet the use of the operating system and logs .

Note that server selection is not a waterfall process , The hard disk should be considered comprehensively ,CPU, Memory , network card , In combination with market conditions , Choose what meets your needs 、 Cost effective server configuration .

2） Storage architecture design

A typical Swift The cluster topology is shown in the following figure .

Typically , Nodes can assume one of the following three service combinations ：

• Proxy,Account,Container （PAC）
• Object Services（O）
• All Services（Proxy,Account,Container,Object）（PACO）

Small scale deployment （ Effective capacity <=1PB) when , It is generally used PACO The way ; Large scale deployment （ Effective capacity exceeds 10PB） when , Generally, it is considered to adopt PAC and O At different nodes ; Medium scale deployment （ The effective capacity is greater than 1PB And less than 10PB） when , The service distribution can be determined according to the specific business requirements and future scalability .

in addition ,Swift It allows a cluster to be divided into different domains according to different physical locations （Region）, It can be used for remote disaster recovery （ Domain A Contains two copies , Domain B Include a copy ） And cross domain access （ The data copies are evenly distributed in each domain ）. because Swift Provide read affinity and write affinity , So for cross domain clusters , Reading and writing will be done in the local domain as much as possible , This reduces the impact on performance .

System tuning

One Swift The read-write process concerns the server 、 Hard disk 、 kernel 、 file system 、 cache 、 Network and Swift And other related services , Therefore, performance tuning is a systematic project . Only typical configurations are described here ：

• Select the appropriate block size . The block sizes that can be adjusted include ： Client block size read and written by proxy server , The block size of the object when the proxy server reads and writes , The disk block size read and written by the object server and the block size read and written by the object server from the proxy server .
• The data disk adopts JBOD Pattern , Do not use RAID
• Object Service workers=auto
• Other background processes , Mainly based on server load and demand , adjustment interval, The size of the task processed each time , Concurrent number, etc .
• The network is set to Jumbo Frame

Design and build a Swift Clustering is a complex task , Server involved , Hard disk , The Internet , Topology planning , performance tuning , Demand analysis and other aspects , It needs to be considered comprehensively . in addition , After the system is set up , The system still needs to be continuously monitored , Operation and optimization , And make timely adjustments for future needs .

Reference material ：

https://www.swiftstack.com/docs/admin/hardware.html

《OpenStack Swift: Using, Adminstering and Developing for Swift Object Storage》 Joe Arnold

https://blog.csdn.net/mirale/article/details/22049219