Performance comparison | FASS iSCSI vs nvme/tcp

front love return Gu

In the last article 《iSCSI vs iSER vs NVMe-TCP vs NVMe-RDMA 》 In, we introduced the avenue cloud travel FASS Several storage protocols supported by full flash distributed storage system , Now we will make a series of comparisons , The first is the most widely used iSCSI Agreements and more recent NVMe over TCP（ Abbreviation for NVMe/TCP）.

1 Introduction to test system

01

We used a set based on Intel Processor test platform ,FASS The storage system is based on a two-way Xeon Gold 6154 Processor's server , Models for Intel Server Systems R2208WFTZS, Each processor has 18 Core （ Our configuration turns off hyper threading ）, The whole machine is matched 192GiB DDR4-2666 R-ECC Memory , The machine architecture is shown in the figure below ：

02

FASS The storage system uses a single chip Intel Optane SSD DC P4800X, The raw capacity is 750GB, be based on PCIe NVMe 3.0 x4 Interface .Intel Optane SSD DC P4800X It is a solid-state disk based on new storage media , Its Optane The storage medium is a kind of phase change storage , With the current based on Flash Of SSD comparison , With better performance . We choose it as FASS The storage medium of the system is mainly because its read-write performance is relatively consistent , And it has stable performance . In the system FASS Set up 64GiB Large page memory , And configured a 100GiB Thick configuration volume of capacity as the test target .

Test client based on virtualized environment , The foundation is that one is equipped with four channels Intel Xeon Platinum 8180 Processor's server , Each processor has 28 Core （ When testing, the hyper thread opens ）, The whole system is 4S112C224T（4 processor 112 The core 224 Threads ）, With 768GiB DDR4-2666 R-ECC Memory . The virtualized environment is Windows Server 2022 Datacenter Provided Hyper-V, Virtual machine used 16 individual vCPU,16GiB Memory .

03

There is another important component in the test ： network card , We used two Chelsio T62100-LP-CR Ethernet card , There are two 100Gbps port （QSFP28 Interface ）, Test the client with FASS The system is connected by direct connection , So as to avoid the influence of other network components . Testing the client and FASS We have installed the latest Windows drive （6.16.1.1） and Linux drive （3.16.0.1）. The test is based on the operating systems at both ends , Used the largest Jumbo Frame It may be worth ：9600.

We currently use Ubuntu 22.04 LTS As the operating system of the test virtual machine , As of the test , The latest update Linux The kernel version is 5.15.0-40. comparison Ubuntu 20.04,Ubuntu 22.04 LTS Can better support NVMe/TCP, stay Ubuntu 20.04 On ,NVMe/TCP Occasionally, there will be some problems in application .

The kernel used by the operating system is also built-in Chelsio T62100-LP-CR after SRIOV Technology provides Virtual Function Driver support of virtual network card , meanwhile Ubuntu 22.04 LTS It also carries popular fio Store test tools Of 3.28 edition . Our configured FASS The system storage volume is a single link , however ,Chelsio T62100-LP-CR Of SRIOV Virtual Function Provides 8 individual RSS Queue, They can significantly improve network performance in virtual machines .

2 Introduction to test results

01

We first tested the most commonly used libaio Performance of asynchronous engine , First of all iSCSI Performance diagram of ：

02

Then there is the comparison NVMe/TCP Performance chart ：

stay iSCSI Next ,4KiB The maximum value of random reading has reached 103.984kIOps, In the single fio Threads and 32 When the queue depth reaches , Again ,4KiB Write the highest value randomly 76.646kIOps Also in the single fio Threads and 32 When the queue depth reaches . stay NVMe/TCP Next ,4KiB The maximum value of random reading has reached 373.797kIOps, In double fio Threads and 256 When the queue depth reaches , Again ,4KiB Write the highest value randomly 257.361kIOps Also in double fio Threads and 256 When the queue depth reaches .

03

The test shows that ,iSCSI Next , Multithreading fio With the same total queue depth , Lower performance , The reason is shown in the figure below ：

Ubuntu 22.04 LTS Of iSCSI The components are very new , But it still has the limitation of single process , meanwhile , Multithreading fio When testing, it actually uses only one queue , therefore , Multithreading fio It will actually reduce iSCSI The overall efficiency of , similarly , exceed 256 The depth of the queue will also reduce efficiency , because SCSI The protocol theory supports that the queue depth is 256. On the principle of ,iSCSI It doesn't mean a single queue , But software iSCSI Implementation may often be so .

NVMe/TCP Is different , Under multithreading, the corresponding performance will be significantly improved ：

actually , All in all 5 individual NVMe/TCP queue , Mapped to 5 A network card RSS Queue On , Correspondingly , And it started 5 individual kworker Threads （ In the picture above kworker/0:0H+nvme_tcp_wq Led by ）. It's worth mentioning that ,NVMe Also support than SCSI Deeper queue depth , Up to 64k（65536）, Therefore, the overall performance will improve with the increase of queue depth .

04

Next is 4KiB Delay performance in random read and random write , First of all iSCSI：

Next is NVMe/TCP：

4KiB Random read and random write , Based on point-to-point communication , As mentioned earlier ,iSCSI The average delay can be as low as 126.79us（ Read ） and 193.47us（ write in ）, And in the NVMe/TCP Next , The minimum delay can reach 66.09us（ Read ） and 186.36us（ write in ）. Some exceptions during writing are Chelsio TOE Characteristics lead to .

05

Next is 1MiB Continuous reading of block size / Write performance , First of all iSCSI：

Next is NVMe/TCP：

stay iSCSI Next ,1MiB The performance also reflects the limitation of queue depth mentioned above , The reading performance is 4 Queue depth is the best ：2081.13 IOps, Write performance slowly increases as the queue depth increases , Up to 2248.88 IOps, That means 2248.88 MiBps（2.196 GiBps）, Actually and approaching Intel Optane SSD DC P4800X Nominal limit of （ The official nominal writing is 2200 MBps）.
And in the NVMe/TCP Next ,1MiB The reading performance can maintain a stable level at different queue depths ： about 2500 IOps, Writing is about 1650 IOps about .

3

Test summary

The test results clearly show , Pure software iSCSI and NVMe/TCP The scheme has significant performance differences , One side ,NVMe/TCP With a more streamlined data path （ As shown in the figure below ）, On the other hand ,NVMe/TCP Native supports more queues and greater queue depth , The former reduces latency , The latter improves the overall throughput .

NVMe/TCP Is for NVMe New agreement for flash disk development , In addition to giving full play to the advantages of network cards with multiple queues, many new Linux Kernel characteristic , Such as Zero-Copy etc. , Therefore, it also has requirements for the underlying kernel version —— Mainline required 5.0 Version and above , Compatibility is not as good as being supported by an operating system iSCSI, But as the NVMe Solid state drives are becoming more and more popular ,NVMe/TCP It will also become more and more popular , And gradually become iSCSI An alternative , For not being able to use RDMA Scene , We recommend FASS The user choice of NVMe/TCP agreement .