Development of trading system (VIII) -- Construction of low delay network

Reprinted from 《 Trading technology frontier 》 The thirty third issue of the article （2018 year 12 month ）

One 、 Low latency trading

1、 Introduction to low latency trading

Low latency trading is a branch of algorithmic trading , Capital market institutions react more quickly to market events , Take advantage of extremely subtle time differences , To gain greater trading profitability .

2、 Transaction delay classification

Delay is when a computer system receives an event stimulus , Time interval between and response generation . For brokers , The event stimulus can be the client input order , Can receive market data release , It can be returned after receiving the order confirmation . Low delay trading requires all links in the whole trading chain , Try to shorten the time interval . From the perspective of trading system , Transaction delay mainly includes network delay 、 Protocol delay 、 Operating system latency 、 Application delay, etc . 

3、 Network delay

The order placed by the trading system is delivered to the exchange's trading matching host through the network through the brokerage counter , It will pass through multiple network devices , Including switches 、 Routers and firewalls , So there will be delays in the network .

There are three types of delays in the network ： Data serialization delay 、 Transmission delay 、 Queue delay .

（1） Data serialization delay

The data serialization delay is that the network device transfers a certain amount of data bits （bit） Enter the physical media （ It usually refers to optical fiber or copper cable ） The time required .

（2） Transmission delay

Transmission delay transmission delay （propagation） After the data is serialized into the transmission medium , The time taken for transmission in the transmission medium . The transmission speed of light in optical fiber is usually two-thirds of that in vacuum .

（3） Queue delay

When multiple data sending ends send data packets to one receiving end through the same network link , Packets must be queued on the switching equipment connecting the sending end and the receiving end （queuing）. The size of a packet queuing delay depends on two factors , One is the number of packets at the top , One is the bandwidth rate of the data receiving link . 

4、 Protocol delay

There are many network transmission protocols in the trading network , such as TCP、UDP、SPX etc. , Because different protocols are designed for different purposes , Therefore, the transmission efficiency and the corresponding delay are different . For network protocol delay , The following features need to be considered ：

（1） Nagel algorithm （Nagle Algorithm）

（2） Data window （Data Window） Run out of

（3） The bag is missing （Packet Loss）

（4） Routing protocol

（5）QOS And SPAN Use

5、 Operating system latency

Between the computer hardware and the trading system , There is BIOS/FIRMWARE、 Operating system kernel 、 Code running environment （ Included in the operating system ）, Provide various function calls for the trading system , To realize the interaction between code and hardware , There is also a certain delay in the calling process of the operating system kernel and code running environment .
The equipment manufacturer generally recommends that BIOS and Firmware Update to the latest version and make special settings , And provide solutions or best practices on low latency , This includes providing configuration lists and scripts , It is used to check whether the basic software settings at all levels comply with the best practices of low latency systems, etc .

6、 Application delay

Low latency applications are generally controlled by the trading institution itself , According to the essential characteristics of general trading applications , Application delay is divided into ：

（1） disk access

Low latency applications use memory databases or solid-state storage to reduce IO Delay .

（2） Interprocess communication

Low latency applications use RDMA And other technologies to reduce the cost of interprocess communication .

（3） Data calculation

Low latency applications use FPGA、GPU And other technologies to accelerate the data calculation process .

（4） Data lock and other shared resource access wait

Non blocking data management system can be adopted , Or smaller granularity of resource management to reduce access waiting .

7、 A measure of delay

Due to the delay of low delay trading system , In each link, it is close to the sub microsecond level , When measuring , Consider not only the average delay as a whole 、 Delay jitter and throughput , And pay attention to the time accuracy of the measurement means and the tool itself .

（1） Average delay

The average delay is the average time that a message travels from one point to another , The lower the better .

（2） Delay jitter

Delay jitter is an important index to measure the predictability of delay , It represents the measurement process , The degree of deviation between each actual delay and the average delay ..

（3） throughput .

Throughput can be defined as the amount of data processed per unit time . In general, low delay systems require that low delay be maintained , Pursue the maximum throughput .

（4） Time precision

Most timing systems rely on hardware such as silicon crystal oscillation and motherboard interrupt , Different operating system platforms adopt different precision control methods . Some low latency network cards and switches can provide high-precision time stamps for delay measurement .

Two 、 Introduction to low latency networks

1、 Introduction to low latency networks

In recent years , The financial and capital markets continue to develop at a high speed driven by business and technology , Global exchanges tend to be centralized, integrated and interconnected , Trading products and business diversification and innovation . Information technology innovation is changing with each passing day , Promote the continuous upgrading of the trading system , It is the trend of the development of trading technology in the domestic securities industry that the trading system changes from centralized to distributed .

The trading bidding principle of the securities trading system is price priority 、 Time first , That is, at the same price , First come first served basis . At present, the entrusted uplink delay of the centralized trading system of the domestic mainstream securities companies ranges from tens of milliseconds to hundreds of milliseconds , In the face of increasingly fierce competition , The new generation of trading system with distributed architecture has moved towards a delay of 100 microseconds , Time delay is the most important index to measure the performance of trading system .

Global exchanges measure and compete with microsecond delay . To meet market requirements , Adopt advanced high-performance network equipment and technology , Build high reliability 、 Low latency 、 The demand of high throughput network is becoming more and more urgent .

Every year, 6 The month and 11 month ,Top500 Supercomputer The website announces the supercomputer world 500 Strong list 、 Performance indicators 、 Technical structure, etc , Among them, network interconnection technology is the benchmark of high-performance network technology .2018 year 11 month , Before the supercomputer 500 The strong list shows , The most popular interconnection technology is Ethernet （252 A system ）, And then there was InfiniBand（135 A system ）, Custom interconnect （64 A system ） and Intel Omni-Path（43 A system ）

 

Yes 2016 - 2018 year Top500 Statistics on the trend of the network interconnection technology of supercomputers

 

Application and development trend of high performance network technology ：

（1） Ethernet and InfiniBand Are two main network protocols , 10 Gigabit Ethernet is the most widely used system interconnection technology ;

（2） In the past two years , With the rapid development of converged Ethernet technology and the significant improvement of Ethernet bandwidth , The share of Ethernet has gradually expanded to 51%, Get close to InfiniBand Share of 2 times ;

（3）Intel main Omni-Path High performance computing architecture , Provide switches 、 Management software 、HFI card 、 Cable and other integrated solutions .Omni-Path As InfiniBand The main challenger in the field of supercomputing and high-performance networks , from 2016 year 11 Month so far , use Omni-Path The number of systems in shows a gentle upward trend , At present, the overall proportion is 9%.

2、 Ethernet

Ethernet is in 1970 Launched in the early s , With simplicity 、 Easy to use 、 economic 、 Easy to expand and other advantages are widely used . so far 40 For years , Ethernet technology and protocols are still evolving , Meet the needs of today's high-performance network convergence innovation . In recent years , To meet Cloud Computing 、 Unified storage and other diversified businesses , The new Ethernet is through DCB、VXLAN、FCoE Technology , Rich storage and virtualization support capabilities ; adopt RoCE、 Congestion control optimization 、 Flow control management optimization and other technologies , Greatly reduce the network forwarding delay . Ethernet bandwidth has changed from the early 10M Speed up to 10G/25G/40G/50G/100G, Recently, the Ethernet bandwidth has been further increased to 400G, Meet the bandwidth intensive needs of large-scale data centers and cloud computing .

The rapid development of High Performance Ethernet technology , Make domestic 、 Exchanges in major developed markets at home and abroad actively invest resources to upgrade and update network infrastructure , Such as the New York Stock Exchange (NYSE)、 Nasdaq Stock Exchange (NASDAQ)、 The London Stock Exchange (LSE)、 Tokyo Stock Exchange (TSE)、 Hong Kong Stock Exchange (HKEX) All adopt 10 Gigabit Ethernet , And provide 10 Gigabit managed computer room access .

3、InfiniBand

InfiniBand stay 1999 First proposed in , Is a large-scale 、 Network technology designed for high performance server cluster with low delay , The server is realized through a large bandwidth link 、 Storage device interconnection . One of its features is the connection of super wide bandwidth . Bandwidth support 56Gb/s、100Gb/s And 200Gb/s; Second, extremely low network delay .InfiniBand It has lower delay than Ethernet at comparable data rate ; Third, a reliable and lossless network . Flow control mechanism based on link layer and advanced congestion control mechanism prevent congestion and packet loss ; Four is complete CPU Uninstall function . The hardware based transmission protocol achieves high reliability 、 Network data transmission without packet loss . Through remote direct memory access (RDMA) And kernel bypass technology , It can directly read and write the pre allocated memory between remote nodes , There is no need to involve the operating system kernel 、 No need to replicate data multiple times , Significantly lower CPU Workload and network latency . V. excellent scalability and flexibility . A subnet can support 48,000 Nodes , One network can support the deployment of large-scale servers .

A complete set InfiniBand The system consists of a server 、 Switch 、 Subnet manager 、 Host channel adapter (HCA) form . The most widely used network architecture for high-performance server clusters is fat tree （Fat-Tree）, Non blocking network , This architecture can achieve the optimal performance of large-scale servers . A typical fat tree architecture is shown below ：

 

The internal communication of some trading systems is to achieve the minimum delay , The core network adopts InfiniBand, Other networks adopt Ethernet , For example, the trading system intranet of the German Stock Exchange and the Singapore stock exchange adopts InfiniBand, German Stock Exchange T7 trading system 6.1 The version entrustment return loop delay is less than 52us, The average order return delay of SGX is 90us.

3、 ... and 、 Low latency network construction

1、 Low latency factor

The end-to-end delay of the system is mainly transmitted by the network equipment 、 Transmission delay of network connection 、 Network card data receiving and sending delay 、 Operating system processing delay 、 Middleware processing delay and business processing delay etc . The network delay is transmitted by the network device 、 The transmission delay of the network connection and the data receiving and sending delay of the network card , Build a low latency network , It is necessary to reduce delay loss in all links .

2、 Forwarding delay compression

With the industry's low latency 、 High performance transaction scenarios and data center hosting business development requirements , More stringent requirements for the performance of network equipment . Regardless of packet size and traffic type , Stable low delay transmission shall be provided between all network ports , And keep the delay jitter low enough .2012 year 9 month , Cisco released the self-developed chip and algorithm Nexus3548 Switch , The normal path forwarding delay is 250 nanosecond , Jitter in 5 About nanoseconds .2013 year ,Metamako The company has released ultra-low delay 、 Configurable Layer-1 switch MetaConnect, The delay is only 4 nanosecond , Such a low delay speed means that users can gain the greatest competitive advantage .2014 year ,Exablaze company ExaLINK Fusion Switch , By streamlining functionality and protocols and integrating Xilinx Ultrascale FPGA, Layer 2 Forwarding delay is 110 nanosecond , Layer 1 switching delay is 5 nanosecond .Arista、Juniper、 Huawei and other network manufacturers mostly adopt Broadcom、Intel Commercial ASIC Chip solutions , According to the data on the official website ,Arista 7150s-52 The delay of is 380 nanosecond ,Juniper Of QFX 5110 The delay of is 550 nanosecond .Mellanox Using self-developed Spectrum chip , Ethernet switch SN2410 The delay of is 300 nanosecond ,InfiniBand SX6036 FDR(56Gb/s) The delay of the switch is 200 nanosecond ,InfiniBand SB7800 EDR(100Gb/s) The delay of the switch is 90 nanosecond .

The competition for low delay network equipment is extremely fierce , The forwarding delay of ultra-low delay switch has gradually approached the physical limit of optical transmission speed . actually , The service processing delay and system overhead are far greater than the network device delay , Forwarding delay of network equipment 、 Performance and other indicators can be used as a reference for comparison . Especially in the core trading system , First of all, we should pay attention to the highly available network architecture and the performance of the network in complex fault scenarios , The second is to reduce the delay as much as possible .

3、 Network architecture

The traditional network architecture of large data centers , It is usually divided into core layer 、 Convergence layer 、 Access layer , With the cloud data center 、 Distributed computing is booming , There is a surge in server interaction traffic in the data center , The data flow model changes from north-south to east-west . The traditional network architecture is designed for north-south traffic , It can no longer meet the requirements of high performance 、 Requirements for low delay distributed system deployment . at present , The method of constructing low delay network architecture is to simplify the layers of traditional network as much as possible , Reduce the number of times data passes through network devices , The traditional centralized exchange 、 The vertical network architecture becomes horizontal 、 Scalable distributed architecture .

Classify according to the number of servers , Switch press 48 Access port calculation , A data center is smaller than 40 The low latency network architecture of a server usually has only one layer （ Core layer ）, The two switches are interconnected as core switches , Also as an access switch . Running between switches M-LAG or vPC Or stack , Make sure no loops are created . Same as data center 1 Skip the network to reach , Across data centers 2 Skip the network to reach , The network architecture is shown below ：

 

One data center is more than 40 The low latency network architecture of a server usually has two layers （ Core layer 、 Access layer ）, The two switches are interconnected as core switches , Other switches are used as access switches . Running between core switches M-LAG Or stack , Make sure no loops are created . Same as data center 1 Skip the network to reach , Across data centers 4 Skip the network to reach , The network architecture is shown below ：

 

4、 Transmission delay optimization

Network transmission delay is the delay caused by data transmission on the network link , Network transmission delay and system deployment location 、 The length of long-distance lines is closely related to the transmission medium , Select the shortest path and the best quality link , It can ensure that the network transmission delay of the link is minimized .

To facilitate the flexibility of market participants 、 Fast access , The exchange provides a custody machine room 、 A variety of access methods, such as data leased lines and even the Internet . The technical infrastructure and system deployment location are connected to the nearest exchange , Deploying the system in the trusteeship computer room of the exchange can effectively shorten the network access delay , It is conducive to building a fast trading channel with point-to-point direct connection to the exchange . To ensure fairness , The exchange provides almost exactly the same length of cabling in the custody room 、 And unified bandwidth access port . Market participants to obtain the minimum network communication delay , LAN with low delay shall be built , The high-speed optical fiber link shall be used as much as possible for intra city and inter data center communication , Use as few repeaters as possible on the communication link 、 The connector 、 A firewall 、 Routers and other devices .

5、 Low latency 10 Gigabit network card

The forwarding delay of low delay network equipment is in the order of hundreds of nanoseconds , The transmission delay of the internal network connection in the same data center is also in the order of hundreds of nanoseconds , by comparison , Network card data receiving and sending delay accounts for the main proportion . The following table shows the delay reference values of several mainstream network cards ：

 

Even if it is the same 10 Gigabit network card , The performance of different series of network cards is very different , The low delay 10 Gigabit transmission delay is half that of an ordinary 10 Gigabit network card . The network card data receiving and sending delay accounts for such a high proportion , Therefore, in the process of building a low delay network , It is necessary to select a low latency 10 Gigabit network card .

6、 Network bandwidth improvement

Serialization delay refers to the time when a data frame or cell is received by a receiver and starts processing . The serial delay is inversely proportional to the transmission rate , The higher the speed , The shorter the time it takes to receive a data frame . Improving network access bandwidth is the most effective way to reduce serial delay , The lowest recommended access bandwidth for low delay networks is 10G, Recommend 25G Or higher access bandwidth .

At this stage , Most data centers are based on 10G Ethernet planning and construction . With 25G The cost of switches and network cards is greatly reduced , at present 25G The total cost of ownership is basically close to 10G, The network cabling can be upgraded to... Without any change 25G bandwidth . according to Dell'Oro Group forecast ： With support 25G The number of network switches and network cards is increasing , More cloud users or large enterprises upgrade to 25G.2019 - 2020 year ,25 G Will become the mainstream speed of cloud and enterprise servers ,25 G Port shipments are expected to be in 2021 Peaked in .

Sinovel distributed technology laboratory cooperates with the low delay Technology Laboratory of Shanghai stock exchange technology company , Use Qperf For a mainstream brand of low latency 10 Gigabit network card in 10G and 25G Model for TCP、UDP Time delay benchmarking , The analysis and comparison results show that ：25G Various packet lengths TCP and UDP Time delay , All are lower than 10G Delay of .

  

Even if the bandwidth occupied by the network card port is far from 10Gb/s The level of （ Like a hundred megabytes b/s The bandwidth of traffic ）, The above test results are still valid . To further analyze the impact of bandwidth on the performance of the application system , Sinovel's ATP The trading system is in the same test environment , Test based on different bandwidth modes of the network card , It turns out that , The system delay is reduced by more than 20% Of , The processing performance of the application has also been improved .

7、 Network card acceleration

The network card is an important part of the server , The performance of the network card will directly affect the overall performance of the system . Through network card acceleration technology , It can greatly and effectively reduce the overall delay of the system . In recent years , Ethernet continues to integrate innovation , Development launch RoCE agreement .RoCE Support standard Ethernet protocol at data link layer , Support... On the network layer IP agreement ; There is no need to change the original infrastructure of the data center , Significant cost savings ; It also makes the performance of large bandwidth Ethernet fast to InfiniBand close .

except RoCE Network card acceleration technology , Acceleration technology is commonly used in the industry , High performance POSIX Compatible with network stack , It can significantly reduce network delay 、 Improve throughput and reduce CPU utilization . Provide standard BSD API supports Ethernet network protocol , End users do not need to modify the application . Technology bypasses through the kernel 、 Reducing the number of data copies and other technologies will make the traditional kernel stack about 7us The delay is reduced to less than 2us.

8、 Lossless network

The traditional Ethernet adopts the best effort forwarding mode , It is sensitive to time delay 、 Performance such as packet loss rate does not provide any guarantee . When multiple servers in the network communicate with one server at the same time , It often causes network data congestion and even packet loss . Network congestion and packet loss lead to retransmission , Increase network delay , In severe cases, the business performance will also be affected . The common solution to reduce or avoid network congestion and packet loss is ：PFC(Priority-Based Flow Control ) and ECN(Explicit Congestion Notification ) Flow control technology .PFC You can control the flow of queues on the link , In case of congestion at the switch entrance, the upstream equipment flow is backpressed .ECN Technology can realize end-to-end congestion control , When the switch outlet is congested , Do... On packets ECN Mark , And let the traffic sender reduce the transmission rate . By combining deployments in the network PFC、ECN Function to realize lossless transmission guarantee , Ensure that the network keeps running with low delay .