Implementation of aut, a self-developed transport layer protocol for sound network -- dev for dev column

This paper is about 「Dev for Dev special column 」 Series content , The author is the head of the large back-end transmission protocol of the sound network summer .

For the new demands and challenges brought by real-time interactive applications to network transmission , The voice network layer and decouple the application layer business requirements and transmission strategies in real-time interaction , On 2019 Self developed internal private transport layer protocol Agora Universal Transport（AUT）, Bring together various transmission control capabilities under heterogeneous networks , And in 2021 ~ 2022 Since, it has been gradually implemented in various businesses on a large scale , Using a set of transport protocols / The framework solves the different transmission requirements of various services .

The relevant content is divided into two parts , This article will introduce in detail AUT The design and evolution of transmission protocol .

01 Complex transmission scenarios

As a company that provides a real-time interactive platform , Transmission is undoubtedly the cornerstone of all interactions , With the development of the Internet , The interactive scenes are becoming more and more diverse , The content of transmission is becoming more and more complex , Transmission under real-time interaction mainly faces the following requirements and challenges ：

Media data transmission

RTC The main real-time interactive scene of the sound network , The transmission of real-time audio and video is also the transmission scenario most closely coupled with the upper business logic , The transmission logic needs to work closely with the source coding at the media level to achieve low latency interaction .

● A reliable network channel is required , To control the sending and receiving of messages .

● Multiple reliable real-time channels are required , To meet multi-channel data flow （ Audio and video etc. ） Send and receive .

● When bandwidth is limited , The priority management of the above flows needs to be solved （ Control channels > Audio > video ）.

● To B scenario , The customer should be able to decide the priority of the flow and the transmission degradation strategy independently and flexibly .

● Relevant strategies in the media , Need to work closely with the current network conditions .

● When transmitting in the network , The network quality of interworking between different regions and operators varies greatly , All kinds of networks buffer、 There is a huge difference between delay and packet loss .

Universal data transmission

Sound network launched FPA Full link acceleration The products are used in various applications all over the world （ Include Web/ game / Audio and video etc. ） Provide end-to-end link acceleration services .

● It supports both end-to-end reliable and unreliable transmission channels .

● As a universal data acceleration channel , The data flow varies widely .

● Low latency .

Low latency reliable message transmission

RTM It is stable and reliable provided by the sound network 、 Ultra low delay 、 Highly concurrent global signaling and message cloud services .

● Small message size , Message frequency is high , The overall flow is relatively low .

● Low latency .

Low priority reliable data transmission

Report It is the embedded point and event data reporting in the sound network , Provide internal and external troubleshooting information .

● Low transmission priority and real-time requirements , Avoid impacting other business data .

● Keep a long connection to the back end , The number of connections is huge , But most of the time is free .

Intra network transmission of various services

SD-RTN It is the global coverage network within the sound network , Cover 200 Multiple countries and regions , Carrying traffic of different services .

● Long fat link in cross region transmission （ That is, Changfei network , Large bandwidth delay product 、 High packet loss rate ） The transmission of , Bandwidth climb with large link delay / Packet loss should be recovered quickly .

● Different countries / It is difficult to guarantee the network quality between operators , Packet loss / Jitter is very common .

● Different businesses / Customer / Different degrees of the scene QoS Guarantee .

02 Solution – Self developed transport layer protocol

Shortcomings of the existing scheme

Transmission requirements vary , First of all review Various products in the industry , In order to obtain mature solutions , However, despite the research findings, there are various unsatisfied places ：

programme	advantage	limited
RTP/RTCP	The transmission strategy can be customized	No priority management ; Lack of multiplexing capability ;
TCP/RTMP	Good versatility	It is easy to block the queue head and cannot be multiplexed ; Unable to customize the transmission policy , There are few optimization schemes ;
SRT	There are some weak network confrontation mechanisms	Lack of independent congestion control ; Lack of multiplexing capability ;
QUIC	With multiplexing 、 Priority management and anti row head blocking	Lack of transmission support for real-time unreliable data streams ; Lack of various weak network countermeasures and network adaptation capabilities ; Large and comprehensive agreement , But it is too heavy for internal use ;

We found out from the business requirements , There are similarities as well as differences in the transmission between services , It is still satisfying to seek the existing solution for the transmission of a service alone , But it is not easy to use a transport framework to take into account all business scenarios , This requires in-depth understanding and abstract refinement of business requirements , Various products in the industry are difficult to realize .

Final , Self developed transport layer protocol / The frame becomes the lightest 、 The most adaptable , But it is also the most challenging solution .

AUT The design goal of

According to the transmission requirements and characteristics of existing services , This paper combs and summarizes several objectives that the self-developed transport protocol needs to achieve , as follows ：

1、 Provide multiplexed transmission channels to simultaneously Carrying different data types , and Provide effective descriptions of different types of data programme .

2、 Provide Sufficient scalability , It can adapt to the transmission related logic and input of various application layers , For example, whole block data / Block data / Scattered data, etc , Real time transmission is often combined with applications to uncover secrets , The granularity of the underlying transmission control will also be finer .

3、 Provide... For different data Different levels of transmission quality assurance , For example, different priorities / reliability / Redundancy .

4、 Flexible transmission control module interface , Scalable implementation of different congestion control 、 Packet loss detection 、 Network detection and other strategies .

5、 The underlying network is interfaced , can Support any virtual network .

6、 On the upper Provide effective network quality analysis , It is convenient for the upper layer to apply different business logic for different network quality .

7、 Adapt to different network scenarios , Adapt to the disorder of heterogeneous networks / shake / Grow fat / Network conditions such as current limiting .

03 AUT The evolution of

Architecture evolution process

● The initial prototype verification phase , We use the simplest model to complete the abstraction of the transport layer , It is only applied in the scenario of unreliable transmission in audio and video media .

● After prototype verification ,AUT Evolve into Session and Connection The layered design of the system ：Session Mainly responsible for Stream Management of , Different Stream Fulfill different business requirements ,Connection Responsible for pure transmission control independent of business , In this way, the decoupling of transmission control and business requirements is completed ; Independently abstract the interfaces related to network transceiver , bring AUT Sure Overlay On various networks ; Introducing encryption / Safety features such as certification ; introduce MTU Probe 、Padding And other network state detection capabilities .

● After completing the layered design ,AUT The layering of ： stay Stream More fine-grained general-purpose modules and special-purpose modules , Through abstraction Writer/Reader 、 Combining stream controller and buffer to complete different channel coding and decoding 、 Flow control and retransmission control , The weak network antagonism capability of each flow is differentiated to adapt to different service data ;Connection Overall management of multiple Path, Every Path As independent transmission control units, they do not affect each other , From being able to support multi tone at the same time Path Transmit at the same time ; In the transmission algorithm , Traffic model detection is introduced 、 More in-depth network state detection modules such as packet loss type detection .

Weak network antagonism

Along with AUT Evolution of , The dimensions of our analysis and evaluation of the network are also gradually enriched , These network analysis capabilities are AUT The foundation of weak network antagonism capability ：

● Disorder detection capability ： Detect the disorder degree in the network , And make corresponding adjustment to the transmission algorithm under the condition of out of order , For example, adjust the packet loss detection algorithm and congestion control algorithm to adapt to the disordered scenario ;

● Bandwidth detection capability ： Bandwidth detection algorithms with different dimensions , Complement each other's advantages under different detection requirements ：packet train Probe bandwidth limit , Fuzzy detection of high transmission bandwidth with very small traffic ;padding Probe actual bandwidth , When the application data is insufficient , Populating the network with actual traffic , Veritably verify the actual capacity of the network ;

● Traffic model detection capability ： Test the model of actual network traffic , For example, typical in the public network traffic policing/shaping Equal flow restriction strategy , And adjust the sending control strategy ;

● Packet loss type detection capability ： Analyze the real packet loss mode , Enter the current packet loss model , For example, random packet loss and congestion packet loss , And dynamically compensate in other modules according to the packet loss type .

After defining the current state of the network ,AUT All kinds of internal weak network antagonism ability can “ An antidote against the disease ”：

● Stream Level universal channel coding capability ：Stream Internal implementation of a general packet codec framework , in the light of FEC The block codes in can be easily integrated , Make a difference Stream Retransmit 、 Flow control and other differences , It also has the protection of different coding strength , Wider external transmission scalability ;

● Dynamic feedback control capability ：AUT The main feedback is Ack package , Effective Ack Feedback guarantees the accuracy of internal algorithm logic ： Dynamic enable AckAck against Ack Packet loss ; Different Ack Delay Adaptive performance / Delay ;Ack Link jitter / Burst detection , Compensation transmission control ;

● Deeply optimized congestion control / Packet loss detection algorithm ： Internal control of typical congestion / Packet loss detection algorithm is deeply optimized and explored in various scenarios , While maintaining the characteristics of the algorithm itself , It is suitable for various transmission control requirements in real-time transmission .

Transmission effect comparison

As a transport layer protocol , We also compared the effects of some real business scenarios with other transport layer protocols , Some results are as follows ：

It can be seen that both high and low bandwidth limits （100Mbps and 4Mbps）、0-50% Random packet loss of the single uplink and simultaneous uplink and downlink 、20ms and 200ms Of RTT scenario ,AUT comparison lsquic It can achieve throughput closer to the actual bandwidth .

Overview of landing scene

Up to now ,AUT Gradually fall into the business scenario below the sound network ：

RTC

AUT In the sound network RTC The overall position in the business is shown in the figure below , You can see , When the user accesses the edge node of the sound network Lastmile, The edge nodes of the harmony network are based on SD-RTNTM Network transmission , All use AUT As 4 Layer transport protocol .

Lastmile Use AUT agreement , Let's reconstruct the collaborative relationship between media and transmission as a whole , Enables faster network state tracking （ The bit rate climbs from 20+ Seconds increase to 3 About seconds ）、 Better audio first experience （ No caton ）, And effectively reduce the jam and delay 、 Better support for hardware coding, which greatly improves the user experience of weak networks .

Under different network conditions , Use AUT Some indicators of the post video are as follows ：

FPA

FPA in AUT Location and RTC similar ,FPA The transmission data in is no longer limited to audio and video , But because of reuse AUT, The transmission capacity in various scenarios has also been greatly improved , We test in different areas FPA The transmission rate with the public network , The relevant data are as follows ：

RTM

In the access RTM In the edge node of ,RTM Use AUT Replace TCP, Use AUT Access RTM The arrival rate and delay under various weak networks are significantly ahead of TCP Access . By sending 1000 Message data and record its arrival time , Contrast test AUT And from TCP The test results of the representative public Internet transmission protocol in three cases show , Use AUT comparison TCP At the speed limit 100Kbps Under this condition, the average message arrival delay decreases 53%, Packet loss 20% The average message arrival delay decreases 67%, The speed limit + Under the condition of packet loss, the average message arrival delay decreases 55%.

Report

We're connecting Report and RTC Edges are used at the same time AUT, And in Report Of AUT Use... In connection LEDBAT Algorithm , If there is a shared bottleneck bandwidth for each connection at this time ,Report The connection can participate in transmission with polar competition , Ensure minimal impact on other business data .

SD-RTN

In the sound network SD-RTN In network use AUT agreement , So that the intra network transmission can provide different QOS The ability of , Further improve the transmission efficiency of business data , Use under the long fat network FEC Reduce the number of retransmissions , Further reduce the intra network delay under the weak network , Of the public network Traffic Policing/Traffic Shaping Can clearly detect and adapt , Avoid packet loss caused by over sending , Network quality assessment is more explicit 、 Better network path planning is realized ,Metadata The mechanism greatly simplifies the control surface logic of the business .

in the light of RTC Intra network transmission results , We took 40 Compare the two computer rooms , give the result as follows ：

programme	AUT Before using	AUT After using
The time when the arrival rate fails to reach the standard	00.00709%	00.00489%
The jitter rate does not reach the standard	00.00577%	00.00564%

Because of the previous RTC The transmission quality within the network has been excellent , The arrival rate and jitter related indicators have been achieved 4 individual 9 Under the premise of reaching the standard , introduce AUT Further reduce the non-compliance time , bring SD-RTN The quality of our products is further in line with that of the special line .

About Dev for Dev

Dev for Dev The full name of the column is Developer for Developer, This column is about the sound network and RTC Developer interactive innovation practice activities jointly initiated by the developer community .

Technology sharing through the perspective of Engineers 、 AC collision 、 Project Co Construction and other forms , Bring together the power of developers , Mining and delivering the most valuable technical content and projects , Fully unleash the creativity of Technology .