Large file transfer software based on UDP protocol

be based on UDP Protocol design of large file transfer software

One 、 Design requirements

1、 The system requires the transmission of large files , Programming language is unlimited , It consists of client and server ;

2、 The client and server can run on the virtual machine or development board , A file is created every minute on the client and named with a timestamp , The default size of each file is 6GB;

3、 Client pass UDP The protocol repeatedly sends files to the server , The file transfer rate is required to reach 10MB/s, requirement UDP The file transmission packet loss rate of the protocol is less than 5%, Automatically delete after file transfer ;

4、 The server accepts and stores the file , Total storage 20 minute , Each file is required to store only 1 minute . exceed 20 Minutes of files are automatically deleted ;

5、 The server stores the file , Packet loss rate of file transfer , Transfer rate and other information are written to the log file ;

6、 The server provides a graphical operation interface , Users can view the files and status information stored in the current server . You can also manually select file transfer to the client .

Two 、 Design summary

Software programming language C++、 Frame usage QT, Design a file transfer client and a file transfer server .

The network transmission protocol adopts UDP,UDP Provides connectionless communication , And there is no reliability guarantee for the transmission of data packets , Suitable for transmitting a small amount of data at a time ,UDP The application layer is responsible for the reliability of the transmission .

UDP The protocol uses the check value in the header to ensure the security of data . The check value is first calculated by a special algorithm at the data sender , After delivery to the receiver , You need to recalculate . If a datagram is tampered by a third party during transmission or damaged due to line noise, etc , The sending and receiving verification calculations will not match , thus UDP The protocol can detect whether there is an error . This is related to TCP The agreements are different , The latter requires a check value .

UDP It's a connectionless protocol , No connection between source and terminal is established before data transmission , When it wants to transfer, it simply grabs the data from the application , And throw it on the Internet as soon as possible . At the sending end ,UDP The speed of data transmission is only affected by the speed of data generated by the application 、 The capacity of the computer and the limitation of the transmission bandwidth ; At the receiving end ,UDP Put each message segment in the queue , The application reads one message segment at a time from the queue .

Because the transmission data does not establish a connection , So there is no need to maintain the connection state , Including sending and receiving status, etc , Therefore, a server can transmit the same message to multiple clients at the same time .

UDP The title of the packet is very short , Only 8 Bytes , be relative to TCP Of 20 In terms of a byte packet UDP The extra cost is small .

Throughput is not regulated by congestion control algorithm , Only affected by the rate of data generated by the application 、 Transmission bandwidth 、 Source and terminal host performance limitations .

UDP It's message oriented . The sender's UDP Message delivered to the application , After adding the header, deliver it down to IP layer . Neither split , It doesn't merge , But keep the boundaries of these messages , therefore , The application needs to select the appropriate message size .

In order to transmit data more conveniently and stably in program design 、 Adopted UDT Data transmission application layer protocol .

UDT Is based on UDP The data transfer protocol （UDP-based Data Transfer Protocol, abbreviation UDT） Is an Internet data transmission protocol ,UDT The main purpose of is to support massive data transmission over high-speed Wan .

UD Breaking the data transmission bottleneck , It is based on UDP Application level data transfer protocol , For distributed data intensive applications over wide area high-speed networks .UDT Use UDP Transmit bulk data through its own reliability control and congestion control mechanism . The new agreement is comparable to TCP Higher speed data transmission .UDT It is also a highly configurable framework , It can accommodate various congestion control algorithms .

It's characterized by speed 、UDT It is designed for ultra-high speed network , Has been used to support TB Global data transfer of level data sets .UDT Is a lot of commercial WAN Accelerate the core technologies in the product ; concurrent UDT Streams can share the available bandwidth fairly , and UDT Also for the TCP Leave enough bandwidth .UDT Completely at the application level . Users only need to download the software to start using . No kernel reconfiguration required . Besides ,UDT Of API With traditional sockets API Very similar , So you can easily modify existing applications .

UDT User defined congestion control algorithms can be supported through simple configuration . Users can also modify UDT To adapt to all kinds of situations . Students and researchers can also use this function to study new control algorithms .

UDT Based solely on UDP, This makes it easier to traverse the firewall . in addition , Multiple UDT Streams can share a UDP port , So the firewall can only be used for all UDT Connect and open a UDP port .UDT Cross connect settings are also supported .

UDT Software is a C ++ library , It includes UDT API Implementation and programming examples . The latest version is UDT edition 4, Include 3 A separate software package ： Pure source code ,GNU Software packages and precompiled WIN32 / i386 edition . It can be downloaded from UDT SourceForge The project website downloads all UDT edition .

3、 ... and 、 Software finished product drawing

Four 、UDP agreement characteristic

UDP It's the transport layer protocol , The function is in IP The most basic service is added to datagram service ： Reuse and reuse and error detection .

UDP Providing unreliable Services , have TCP Advantages not available ：

UDP There is no connection , There is no time delay needed to establish the connection . In space ,TCP Need to maintain connection status in the end system , A certain amount of expense is required . This connection load includes receive and send cache , Congestion control parameters and parameters of serial number and acknowledgment number .UCP Do not maintain connection status , And don't track these parameters , Low overhead . It has advantages in space and time .

for instance ：

DNS If it runs in TCP Above, not UDP, that DNS It's going to be a lot slower .

HTTP Use TCP instead of UDP, Because for text-based data Web For web pages , Reliability is very important .

The same dedicated application server supports UDP when , It must be able to support more active clients .

UDP The overhead of group header is small ,TCP The first one 20 byte ,UDP The first one 8 byte .

UDP no congestion control , The application layer can better control the data to be sent and the sending time , Congestion control in the network will not affect the sending rate of the host . Some real-time applications require a steady rate of transmission , Can hold Endure the loss of some data , But it is not allowed to have a large delay （ Such as real-time video , Live broadcast, etc ）

UDP Provide the best possible delivery , There is no guarantee of reliable delivery . All the work of maintaining transmission reliability needs users to complete in the application layer . No, TCP The confirmation mechanism of 、 Retransmission mechanism . If it is not transmitted to the opposite end due to network reasons ,UDP It will not return error information to the application layer

UDP It's message oriented , For the messages handed in by the application layer , Add the first one and deliver it to IP layer , Neither merge , It doesn't break up , Keep the boundaries of these messages . Yes IP Layer up UDP User datagram , After removing the first part, it will be delivered to the upper application process intact , The message is indivisible , yes UDP The smallest unit of datagram processing .

Because of this ,UDP Not flexible enough , Can't control the number and number of reads and writes . For example, we need to send 100 Byte message , Let's call... Once sendto The function will send 100 byte , The opposite end also needs to use recvfrom Function to receive at one time 100 byte , You can't use a loop to get 10 Bytes , Get ten of these .

UDP Commonly used one-time transmission of a small amount of data network applications , Such as DNS,SNMP etc. , Because for these applications , If you use TCP, For connection creation , Maintenance and demolition costs a lot .UDP Also used in multimedia applications （ Such as IP Telephone , Real time video conference , Streaming media, etc ） Reliable transmission of data is not important to them ,TCP The congestion control will make them have a big delay , It's also intolerable

UDP The first format of

UDP Datagram is divided into the first part and user data part , Whole UDP The datagram serves as IP The data part of the datagram is encapsulated in IP In the datagram ,UDP The data message structure is shown in the figure ：

UDP The head has 8 Bytes , from 4 Fields make up , Each field is two bytes ,

1. Source port ： Source port number , Use when you need a reply , Set it all when you don't need it 0.

2. Destination port ： Destination port number , You need to use .

3. length ：UDP The length of the datagram （ Including the first and data ） Its minimum value is 8（ Only the first one ）

4. The checksum ： testing UDP Is there any error in the transmission of datagram , If there is a mistake, discard .

This field is optional , When the source host does not want to calculate the checksum , Make the fields all 0.

When the transport layer from IP Layer received UDP The data tell the time , According to the destination port in the first part , hold UDP Datagram through the corresponding port , Submit to application process .

If the receiver UDP It is found that the destination port number in the received message is incorrect （ There is no application process corresponding to the port number 0,）, Just discard the message , And by the ICMP send out “ Port unreachable ” Error message to the other party .

UDP check

When calculating the checksums , Need to be in UDP Data report before adding 12 Pseudo header of byte , The pseudo head is not UDP The real first . Just calculating the checksums , Temporarily added to UDP In front of the datagram , Get a temporary UDP The datagram . The checksum is based on this temporary UDP Calculated by datagram . The pseudo header is neither transmitted down nor submitted up , It's just to calculate the checksums . Such checksums , I have checked UDP The datagram , Right again IP The source of the datagram IP Address and purpose IP The address was checked .

UDP The calculation method and IP The calculation method of the first checksum of the datagram is similar , We use binary inverse operation to sum and then reverse , But the difference is ：IP Verification of the checksum of the datagram IP Datagram and the first , but UDP The check sum of the first part and the data part is checked together .

The sender , First, put all zeros in the checksum field and add the pseudo header , And then put UDP The datagram is seen by many 16 The substrings of bits are connected , if UDP The data part of the datagram is not an even number of bytes , Then add an all zero byte at the end of the data section （ This byte is not sent ）, Next, we calculate these according to the binary inverse code 16 The word and . Write the binary inverse of this sum into the checksum field . At the receiver , I received it UDP The datagram plus the pseudo header （ If not even bytes , You also need to fill in all zeros ） after , Calculate these by binary inversion 16 The word and . When there is no mistake, the result is 1,. Otherwise, there will be an error , The receiver should discard this UDP The datagram .

Be careful ：

1. When checking , if UDP The length of the datagram part is not even bytes , You need to fill in a full 0 byte , But the second byte is the same as the pseudo header , It's not sent .

2. If UDP Check and check out UDP The datagram is wrong , It can be discarded , It can also be delivered to the top , But attach the error report , Tell the top that this is the wrong datagram .

3. Through the fake head , Not only can you check the source port number , Destination port number and UDP The data part of a user datagram , You can also check IP The source of the datagram IP Address and destination address .

This kind of error detection ability is not strong , But it's simple , Fast .

5、 ... and 、 be based on TCP/UDP Application layer protocol of protocol Introduce

be based on TCP The application layer protocols of are ：HTTP、FTP、SMTP、TELNET、SSH

HTTP    HyperText Transfer Protocol（ Hypertext transfer protocol ）    

FTP    File Transfer Protocol ( File transfer protocol )    

SMTP    Simple Mail Transfer Protocol ( Simple mail transfer protocol )

TELNET    Teletype over the Network ( Internet telex )    

SSH Secure Shell

be based on UDP Application layer protocol ：DNS、TFTP（ Simple file transfer protocol ）、SNMP： Simple network management protocol

DNS    Domain Name Service ( Domain name service )

TFTP    Trivial File Transfer Protocol ( Simple file transfer protocol

SNMP    Simple Network Management Protocol ( Simple network management protocol )     adopt UDP port 161 receive , Only Trap Information adoption UDP port 162.

NTP    Network Time Protocol ( Network time protocol )

6、 ... and 、UDT Introduction to transmission protocol

Website links :  https://udt.sourceforge.io/

UDT The official download address of the project source code :  https://sourceforge.net/projects/udt/

1, First UDT What is it? ？

UDT Is based on UDP The data transfer protocol .UDT Open source software , The main purpose is to “TCP Poor transmission performance on high bandwidth and long-distance networks ” The problem of , Support... As fully as possible BDP Massive data transmission on the network .UDT Is to establish and UDP Two way oriented application layer protocol on , A new congestion control algorithm and data reliability control mechanism are introduced . It can not only support reliable data stream transmission （STREAM type TCP） And some reliable datagrams （DGRAM Similar to broadcasting messages on the network ） transmission , It can also be applied to point-to-point technology , Firewall penetration , Multimedia data transmission and other fields .

2, hierarchy

One thing worth noting here is UDT Is based on UDP An application layer protocol , This means that he has inherited UDP In addition to all the advantages you can have ,

It means that it is compatible with most operating systems , This is also for UDT The popularity of provides the possibility .

The figure above can well show UDT Layered architecture of the protocol . Application usage UDT Socket Of API Interface , Just like using the system Socket equally .UDT Provides a Congestion control class (CC).CC Contains a set of necessary user-defined callback functions , To handle different controls event . Applications can be customized , Derived congestion control class .

UDT be located UDP The application layer above . Application through UDT Socket Exchange data . Memory copy to reduce processing time , Around UDT.

The picture above shows UDT The software structure of . The solid line above represents the data flow , The dotted line indicates the control flow .

UDT It's two-way , All entities have the same structure . When data needs to be sent , The sent data is forwarded to Sender The buffer , Then be Sender Sent to the underlying UDP channel. On the other side of the connection ,Receiver From the bottom UDPchannel get data , Forward to Receiver The buffer , Take the data rerank, And check if any datagrams are lost . Besides Receiver Also used to handle control packets . It will update Receiver and Sender Of LostList. And trigger the corresponding event , Such as congestion control .UDT The functions of are encapsulated by the above modules , And by providing API Serve the application .

3,UDT Characteristics of

UDT Its characteristics mainly include the following aspects ：

1） be based on UDP Application layer protocol

2） Connection oriented reliable protocol

3） Duplex protocol

4） Have a new congestion control algorithm , And it has an extensible congestion control framework .

Besides UDT Agreement in high BDP The Internet is relative to TCP The advantages of the agreement , We can use the following points to express ：

1）UDT Is based on UDP agreement , And it is sent by the timer , Unlike tcp Need to wait ack Before the next round of sending

2）UDT Congestion control algorithm , It can achieve rapid growth and preemption of bandwidth in the slow startup phase , And gradually reduce the growth rate when it is close to saturation , Make it stable .

3）UDT Processing algorithm for packet loss , And tolerance to noise links , In the environment with large network fluctuation , It's better than the traditional TCP The agreement is more stable

introduce UDT Why

Standard data transfer protocol on the Internet TCP Poor performance on high bandwidth and long-distance networks , And can not make full use of bandwidth . The main reasons are as follows ：

1） Current tcp The congestion window mechanism can not work well in the environment of high bandwidth and long distance , The congestion window is too small , And the increase is too slow, which directly leads to low throughput , Unable to make full use of bandwidth .

Besides TCP Of AIMD Congestion control algorithm reduces the size of congestion window excessively , But it can not quickly return to the high bit to make full use of bandwidth .

2） current tcp Congestion control algorithm in BDP The network has poor RTT Fairness ,rtt It will affect the growth of congestion window , The less easily accessible links are congested  

The slower the window grows , The slower it is sent , Therefore, the farther the link is, the slower the sending rate will be .

7、 ... and 、QUIC Introduction to transmission protocol

QUIC（Quick UDP Internet Connection） It's based on UDP Low latency Internet transport layer protocol . stay 2016 year 11 International Internet Project Task Force (IETF) For the first time QUIC Working group meeting , It has been widely concerned by the industry . It also means that QUIC Started its standardization process , Become the next generation transport layer protocol .

We know ,TCP/IP Protocol family is the foundation of Internet . The transport layer protocol includes TCP and UDP agreement . And TCP The agreement compares ,UDP Lighter weight , But error checking is also much less . It means UDP Often more efficient （ Do not often communicate with the server to check whether the packets are delivered or in order ）, But the reliability is not as good as TCP. Usually the game 、 Streaming media and VoIP And so on UDP, And web pages 、 mail 、 Most applications such as remote login use TCP.

QUIC Today's application layer and the various requirements faced by the transport layer are well solved , Including handling more connections , Security , And low latency .QUIC Integration includes TCP,TLS,HTTP/2 And so on , But based on UDP transmission .QUIC One of the main goals of is to reduce connection latency , When the client connects to the server for the first time ,QUIC It only needs 1RTT（Round-Trip Time） A reliable and secure connection can be established with a delay of , be relative to TCP+TLS Of 1-3 Time RTT To be faster . After that, the client can cache the encrypted authentication information locally , When you establish a connection with the server again, you can 0-RTT Connection establishment delay .QUIC At the same time, it reuses HTTP/2 Multiplexing function of the protocol （Multiplexing）, But because of QUIC be based on UDP So it's avoided HTTP/2 The end of the thread is blocked （Head-of-Line Blocking） problem . because QUIC be based on UDP, Run in the user domain instead of the system kernel , bring QUIC The protocol can be updated and deployed quickly , So as to solve TCP Difficulties in protocol deployment and update .

Now ,IETF Of QUIC The working group is in charge of QUIC The standardization process of the protocol .IETF Community for QUIC Has shown a high interest in standardization . A preliminary QUIC The protocol version has been used in Google's services as well as Chrome In the browser , And deployed by a few third-party developers . It should be noted that QUIC Our standardization work is completely open ,IETF Everyone in the community can make their own suggestions , Finally determine a best solution . Therefore, the final standardized protocol may be quite different from the version used .

because TCP The retransmission mechanism , As long as a packet is lost, it must be judged to be lost and retransmitted , Lead to the problem of team head blocking , however UDP There is no such restriction . besides , It also has the following characteristics ：

Implemented their own encryption protocol , Through something like TCP Of TFO Mechanism realization 0-RTT, Of course TLS1.3 It's done 0-RTT. Support retransmission and error correction mechanism , There is no need to retransmit if only one packet is lost , Use error correction mechanism to recover lost packets .

Error correction mechanism ： By means of exclusive or , Calculate the XOR value of the outgoing data and send out a package separately , The server finds a packet missing , Calculate lost packets from XOR packets of other packets . When two or more packets are lost, the retransmission mechanism is used , Because it doesn't work out .

be based on UDP, You can go to QUIC The mechanism for maintaining connections in your own logic , No longer identified as a quadruple , But with a 64 The random number of bits is used as ID To mark , and UDP It's disconnected , So when ip Or when the port changes , as long as ID unchanged , There's no need to re-establish the connection .

TCP The flow control is through the sliding window protocol .QUIC The flow control of is also through window_update, To tell the peer the number of bytes it can accept . however QUIC Our windows are adapted to our own multiplexing mechanism , Not only does it control the window on a single connection , And every one of them in a connection steam Control window .

8、 ... and 、UDX transmission agreement Introduce

UDX The protocol is based on UDP Reliable transmission of ,UDP File transfer protocol .

UDX Is based entirely on standards c++ Developed a set of UDP Transport Library , similar TCP, Is a reliable transmission algorithm . The main thing is to give consideration to TCP Reliability and UDP In real time . Another important advantage is the controllability of the algorithm .

UDX The purpose of is to allow developers to better and faster to develop efficient UDP Network application software ,UDX mainly oo To design and develop , Interface SDK The form provides , Provide methods and event patterns , Also provide API WIN32 DLL.activex, Provide static library , Dynamic library version .

UDX Built in file transfer interface , Support UDP transit ,P2P Interface , streaming , Packaged , Greatly simplify the workload of programmers , Provide 20 A number of practical example code . According to its characteristics , Can be used for file transfer ,IM, Video chat , The chat room , Video surveillance, etc , Of course, it can also be used in competitive technology ,rpg Such games in , According to the actual situation .

There are windows Version and linux edition .