[book club issue 13] Chapter 2 notes on the packaging format and coding format of video files

Note reference sources ：https://education.huaweicloud.com/courses/course-v1:HuaweiX+CBUCNXLO005+Self-paced/courseware/96d922bf013841c39aabcb94f210e197/6b6d22c4ae644bc4b8b39c27c9ecd491/

FFmpeg Video packaging format

FFmpeg Supports many encapsulation formats , Include MP4、FLV、MKV、TS And other video packaging formats 、MP3、AAC And other audio packaging formats RTMP、RTSP、HLS And so on network protocol encapsulation format .

# ./configure  --list-muxersa64                     framehash               mpeg1vcd                rmac3                     framemd5                mpeg1video              roqadts                    g722                    mpeg2dvd                rsoadx                     g723_1                  mpeg2svcd               rtpaiff                    g726                    mpeg2video              rtp_mpegtsamr                     g726le                  mpeg2vob                rtspapng                    gif                     mpegts                  sapaptx                    gsm                     mpjpeg                  sbcaptx_hd                 gxf                     mxf                     sccasf                     h261                    mxf_d10                 segafilmasf_stream              h263                    mxf_opatom              segmentass                     h264                    null                    singlejpegast                     hash                    nut                     smjpegau                      hds                     oga                     smoothstreamingavi                     hevc                    ogg                     soxavm2                    hls                     ogv                     spdifavs2                    ico                     oma                     spxbit                     ilbc                    opus                    srtcaf                     image2                  pcm_alaw                stream_segmentcavsvideo               image2pipe              pcm_f32be               supchromaprint             ipod                    pcm_f32le               swfcodec2                  ircam                   pcm_f64be               teecodec2raw               ismv                    pcm_f64le               tg2crc                     ivf                     pcm_mulaw               tgpdash                    jacosub                 pcm_s16be               truehddata                    latm                    pcm_s16le               ttadaud                    lrc                     pcm_s24be               uncodedframecrcdirac                   m4v                     pcm_s24le               vc1dnxhd                   matroska                pcm_s32be               vc1tdts                     matroska_audio          pcm_s32le               vocdv                      md5                     pcm_s8                  w64eac3                    microdvd                pcm_u16be               wavf4v                     mjpeg                   pcm_u16le               webmffmetadata              mkvtimestamp_v2         pcm_u24be               webm_chunkfifo                    mlp                     pcm_u24le               webm_dash_manifestfifo_test               mmf                     pcm_u32be               webpfilmstrip               mov                     pcm_u32le               webvttfits                    mp2                     pcm_u8                  wtvflac                    mp3                     pcm_vidc                wvflv                     mp4                     psp                     yuv4mpegpipeframecrc                mpeg1system             rawvideo

MP4 Package format

MP4 Basically, the encapsulation format can be considered as the most common video file format in daily life and work , Mainly because of its wide applicability , Not only in PC End (windows、mac、linux) Very good support , And on the mobile side (Android、iOS) It can also be played smoothly .

MP4 Format standard

MP4 Format standard is ISO-14496 Part 12 and ISO-14496 Part 14, The specific features are as follows :

1. MP4 The document consists of many Box and FullBox.
2. FullBox yes Box An extension of , stay Header Added in 8 position version Information and 24 Bit flags Information .3． Every Box from Header and Data Two parts .
3. Header It contains the whole Box The length of the size of (size) And type (type).
4. Data It contains actual data , It could be pure data , It can also be a son Box. When Box Medium Data Is a series of sub Box when , This Box Also known as Container( Containers ).

MP4 ftyp Box

ftyp Box, It is usually located at the beginning of the file , Which includes MP4 Types of video files 、 edition 、 Compatible protocols and other information .

MP4 moov Containers

moov The container contains MP4 Media data information of video file , There are two sub containers inside : mvhd and trak, The former defines the file header information , The latter defines... In the media file track Information .
moov Containers are generally generated in by default MP4 End of file , When playing online, you need to load a complete file to open it normally , In order to quickly open the video , Need to put moov Put the container in MP4 In front of the file .

MP4 mdat Box

mdat Box, Used to store the actual data content of media files .

FFmpeg Video coding format

FFmpeg Support many video coding formats , such as MPEG4、H263、H264、H265、VP8、VP9 etc. . Next , Emphasis on the H264 Video encoding format .

H264 Rate control

1. VBR: Variable BitRate, Dynamic bit rate mode , The bit rate can vary with the complexity of the image , Therefore, its coding efficiency is relatively high .
2. CBR: Constant BitRate, Constant bit rate mode , Because the code rate is constant , When there are dramatic changes in the picture ,QP The parameter will increase , The image quality will get worse , When the scene is still , And waste bandwidth . The overall image quality of this mode is unstable .
3. ABR: Average BitRate, Average bit rate mode , yes VBR A derivative form of , Within the specified file size , Static or near static image parts use relatively low traffic , High traffic is used in the complex picture , It can be regarded as VBR and CBR A compromise of .

H264 Of I The frame type

Frames are encoded within frames , Frames represent keyframes , You can understand it as a complete preservation of this frame ;

1. One GOP The first image in the sequence is called IDR Images （ Refresh the image now ),IDR The images are all Ⅰ Frame image , but I Not all frames are IDR frame , Only GOP The first of the sequence 1 individual I Frames are IDR frame .
   2．I frame , Also called intra reference frame , Keep a complete picture .
   3． Use only when decoding I Frame data can be reconstructed into a complete image .
   4．I Frame is generated without reference to other pictures .
   5．I Frames are P The frame and B Frame reference frame （ Its quality directly affects the quality of subsequent frames in the same group ）.
2. I Frames don't need to consider motion vectors .
   7．I The frame occupies the largest amount of data .

H264 Of B The frame type

Bidirectional predictive interpolation coding frames .B Frames are two-way differential frames , That is to say B A frame records the difference between this frame and the preceding and following frames
1．B frame , Also called bidirectional reference frame , That is to say B A frame records the difference between this frame and the preceding and following frames , To decode B frame , Not only to get the previous cache screen , You have to decode the image , Through the superposition of the front and back pictures and the data of this frame, the final picture is obtained .
2．B The compression rate of frames is the highest , But when decoding CPU The utilization rate will be relatively high .
3．B The frame is from the front I or P The frame and the following P Frame to predict .
4. B The frame transmits it with the preceding I or P The frame and the following P Prediction errors between frames and motion vectors .5．B Frame is not a reference frame , It doesn't cause the spread of decoding errors .

H264 Of P The frame type

Forward prediction coding frame .P The frame represents this frame and the previous keyframe （ or P frame ） The difference between , In decoding, the difference defined in this frame is superimposed on the previously cached picture , Generate the final picture .
1．P frame , Also called forward reference frame , It represents this frame and the previous key frame （ or P frame ) The difference between , In decoding, the difference defined in this frame needs to be superimposed on the previous cached picture , Generate the final picture ,Р Frame has no full picture data , Only data different from the picture of the previous frame .
2．P Frames are I The frame is separated by 1~2 Frame encoding frame .
3．P The frame is transmitted with motion compensation I or P Frame difference and motion vector ( Prediction error ).
4．P The frame belongs to the inter frame coding of forward prediction . It only refers to the one closest to it in front of it I The frame or P frame .
5．P The frame can be behind it P Frame reference frame , It can also be before and after B Frame reference frame .
6. P Frames are reference frames , It may cause the spread of decoding errors .
7. P The compression ratio of frames is between I The frame and B Between frames .