Decoding and practice of cmaf Technology

summary ：

In today's live broadcasting industry in full swing , Running a variety of streaming media packaging and transmission protocols , For example, the most widely used in the radio and television industry HLS、 Popular on the internet live broadcast platform RTMP、HTTP-FLV And overseas OTT Widely used in the industry MPEG-DASH. These streaming media encapsulation protocols have their own advantages and disadvantages , such as RTMP、FLV This streaming media protocol , It can meet the requirements of real-time live broadcast scene with low delay , But because the container format is old , In some new coding protocol extensions 、 Encryption scheme support , Unable to meet requirements with new iterations . Another example HLS、MEPG-DASH This file slicing streaming media protocol has applied MPEG-TS or MP4 Container format , Expand in encoder 、 Multi track support 、 Copyright protection has a unique advantage , However, due to the defects of slicing generation and transmission , The high end-to-end delay is always criticized by users . In the face of such a fragmented pattern , A brand new 、 Higher compatibility , The general container format and transmission scheme for the above problems came into being .

1.CMAF Media framework introduction

CMAF, Full name Common Media Application Format, By Microsoft、Apple、MLBAM、Akamai And other industry giants to MPEG Put forward and put forward in 2017 An international standard approved in . seeing the name of a thing one thinks of its function ,CMAF Designed to address media scalability 、 Low transmission delay 、 Comprehensive solutions to common problems such as content cacheability , Reduce the cost of streaming media transmission and improve the user experience as a whole .

First, learn about the media object model CMAF The composition of ：

chart 1.CMAF Data model diagram

It can be seen from the model composition structure in the figure , It can be used CMAF Header、CMAF Segments、CMAF Chunks as well as CMAF Tracks To present a CMAF Resource objects . Next, we will introduce these structures respectively .

CMAF Header：CMAF Header Used to describe each CMAF Track analysis 、 Decoding and reality related configurations , It usually starts with a 'ftyp' Type of box, Contains a 'moov'box, It contains the whole set Of box The serial number of .

chart 2. Contains a IOSBMFF Data segment CMAF Fragment

CMAF Fragment： Pictured 2 in , Every Fragment Usually by a ISOBMFF Section composition , It can be decoded and decrypted independently , When doing chunked Multiple... Can be packed during transmission ISOBMFF paragraph .

chart 3.CMAF track Data frame

CMAF Track： Pictured 3 in , Every track Contains information stored in CMAF Encoded media samples in the specified container , Including audio , Video and subtitles , By a CMAF The header fragment and the subsequent... Containing the media sample CMAF Slice composition .CMAF The sequence contains information stored in CMAF Encoded media samples in the specified container , Including audio , Video and subtitles , Derived from ISO Basic media file format （ISOBMFF）.

chart 4.CMAF Segment structure

CMAF Segement： Pictured 4 in , In a CMAF One or more of the sequences CMAF Fragment Can be packaged into one CMAF Segment, Every Segment Independent resource descriptors can be used for reference and transmission . For more efficient coding , Usually every audio and video Fragment The length is in 2-6s, In order to ensure CMAF The effect of low delay ,CMAF Of Segment The length of usually does not exceed 10-12s.

chart 5.CMAF Chunk Data structure chart

CMAF Chunk： Pictured 5 Shown ,CMAF Chunk It is in the live video encoder , In a CMAF Segment Without complete generation, it can be divided into different blocks for transmission and distribution , In this way, every CMAF Fragment Be able to encode incrementally 、 Transmission and player decoding . It can also meet the transmission and identification of broadcast and multicast .

In addition to understanding the above basic data structure ,CMAF The media model of also defines multiple track Set and adaptive bit rate structure 、 To support multilingualism & Multi video angle or encoder selection set and delay bound data structure 、 many CMAF The sequence is synchronously encoded 、 Decoded reference time data model, etc .

2.CMAF Feature introduction

As a general media packaging format ,CMAF The feature advantage of is very obvious , Compare several commonly used streaming media encapsulation protocols ：

agreement	transport	Video encapsulation format	Data segmentation	Multiple tracks subtitle 、 advertisement	Encryption scheme	Time delay
http-flv	http	flv	Continuous flow	I won't support it	nothing	low
rtmp	tcp	flv/mp4	Continuous flow	I won't support it	nothing	low
hls	http	mpeg-ts	Slicing files	SCTE And other national standards	Support DRM programme	high
mpeg-dash	http	mp4	Slicing files	SCTE And other national standards	Support DRM programme	high
cmaf	http	mp4	Continuous flow	SCTE And other national standards	Support DRM programme	low

surface 1. Comparison of multi protocol features

By comparing the above streaming media encapsulation and transmission protocols , Almost all dimensions cmaf It's all perfect PK rival . The following is about extensibility 、 Security 、 A simple analysis of the low delay and multi bit rate adaptive dimension cmaf Characteristics of .

chart 6. Multi channel selection set

Extensibility ： Pictured 6 Shown , First cmaf have access to track To separate audio 、 video 、 Subtitles, etc , You can also use multiple track To describe different coders or different bit rates , This method can support multiple tracks well 、 Multi bit rate and caption scene requirements .

Security ： about OTT For the video industry , Copyright protection has always been a requirement for standardization ,CMAF Inherited HLS and MPEG-DASH For universal DRM programme （CENC） The ability to support . such as CBC Can I support... In the encryption mode Apple Of FairPlay DRM,CTR In encryption mode , Support Google widevine And Microsoft's fairplay, Basically satisfied 99.9% Hardware device level encryption requirements for .

Low latency ：CMAF hold segment Cut into smaller block units for transmission , First, there is no need to wait segment Fully generated encoding delay , Second, faster request response capability can improve the response speed of the player , As a whole, it ensures that the player can get the latest block within the delay generated by a block , Decoding and playing , This reduces latency .

chart 7. many track Switch sets

Multi rate adaptive ：CMAF Defines interoperable CMAF Media profile . These media profiles define the encoding and encoding rules required for decoding and , And the need to ensure seamless trace switching required for dynamic adaptive flows , Swap sets can be CMAF The boundary of the slice is toggled and relies on alternative CMAF TRACK, Adaptively transmit the same stream at different bit rates and resolutions .

3.CMAF Detailed explanation of low delay characteristics

Carefully analyze the above characteristics , In fact, many features are MPEG-DASH Has been implemented in ,CMAF contrast DASH The main advantages of are low latency , Next, we focus on the implementation principle of low delay ：

chart 8. Slicing takes time 、 Schematic diagram of response analysis

In the traditional file slicing encoder , Delays are often caused by several parts , For example, in order to ensure rapid response , Distribute historical slices or to ensure that the latest slices are transmitted ,hold Live connection , Wait for the latest slice to be generated before distribution .

Analyze the case1, To ensure fast response to the player , Directly distributed historical slices 3, Because the length of the slice is 8s, The first fragment will be accumulated 8s Delay , Add the latest ungenerated in the current encoder 3s Cache data , So the delay of this request is 11s about .

Analyze the case2, In order to reduce the delay ,hold Request for accommodation 5s, Then distribute the latest slices , So the delay is 8s, Although the delay comparison case1 Slightly down , But the user's Qoe Not good , During the period of the latest slice generation , Although the delay is guaranteed to be 8s, But all connections will be hold live 0-8s, The user's first screen experience will be poor .

So in CMAF in , This kind of quilt hold And the big delay will be solved , First of all, it can ensure immediate response at any time , secondly , Even if the current shard has not yet been generated , It can also be used. chunk The encoding method sends the decodeable part of the current film immediately after it has been encoded , So for the case Come on , Ensure immediate response , The delay is also controlled at 3s.

For such a large slice , Real time response , It can ensure that the delay is controlled within 0-8s. In a real application scenario , We can control the fragment length a little bit , such as 4s A piece , Then the overall user delay can be controlled at 0-4s, The first screen can also be guaranteed .

4.CMAF Practice and effect display

About CMAF Application , Tencent cloud live broadcast product has completed the development and deployment of encoder , Cooperate with the live broadcast cdn platform , Our overall design idea is shown in the figure ：

chart 9.CMAF Media processing framework

Reuse the original cloud connection 、 Advantages of transcoding processing , increase CMAF Packager modular , Used for processing CMAF Container packaging and other media processing work , Packed into transportable http chunk Push to http server Distribute to the terminal player for playing . The following is the slice generated by the media processing center of Tencent cloud in Hong Kong, China CMAF Streaming cooperates with Tencent cloud live broadcast cdn The effect of distribution is more common dash Flow rendering ：

chart 10.CMAF And the general MPEG-DASH Effect comparison chart

Test environment description ：

Encoder position ： Cloud live China Hong Kong cluster .

Push flow ： Tencent cloud, Hong Kong, China cvm,ffmpeg File streaming .

Slicing service configuration ： The slice of the package module configuration is 4s One ,3 Pieces are divided into window sizes .

Test location ： Shenzhen, China .

Test the player ：dash.js

effect ： The overall effect is ,CMAF Than ordinary MPEG-DASH Flow reduced 15s Left right delay . Of course , The test results are related to the player's strategy .

Long connection multiplexing optimization ：

In traditional dash or hls In distribution , Short connections are often used to request m3u8 File or ts、mp4 Fragmentation , In order to improve the transmission efficiency , We recommend using HTTP1.1 Long connection characteristics , Reuse TCP Connect , The file index list and the slice request are run separately in 2 One of the TCP Connected to the , On the whole, it can improve the transmission efficiency and reduce the connection load of the cloud server , Improve the overall service performance .

Postscript

A lot of technology , It is relatively simple to understand from the perspective of principle , But the real engineering application to the online , Ensure mass 、 Stable and reliable 、 Low cost and robust services , It is another profound knowledge . We will continue to optimize iterations CMAF Performance of , Strive to bring better audio and video streaming service experience to users .

Reference documents

1.Common Media Application Format for Segmented Media-ISO/IEC JTC1/SC29/WG11 N16186.

2.https://github.com/Dash-Industry-Forum/dash.js/wiki/Low-Latency-streaming.

3.https://github.com/cannonbeach/ott-packager.

4.http://www.streamingmediaglobal.com/Articles/ReadArticle.aspx?ArticleID=135885&PageNum=1.