Audio and video synchronization knowledge points you must pay attention to:

========

# All knowledge of audio and video synchronization ：
1、 Now that you are paying attention to audio and video synchronization , So suppose you have basic background knowledge , such as ： Have computer knowledge ; Programming theory （ Know interrupt 、 cache 、 State machine 、 modular ）; Know a programming language ; know MPEG ISO agreement ; know mp4、mkv、ts And so on ; know h264、hevc、mpeg2、mpeg4 And other video stream formats ; know mp3、aac And other audio stream formats ; know RGB、YUV、YCrCb、WAV And other display and playback formats ; Know the display driver audio driver ; Know the screen refresh rate vsync. If you don't know these, you need to search and learn relevant knowledge when you encounter them in practice .

2、 Unpacking （ from mp4 And other files to separate video stream and audio stream ） Get the video frame rate when （ Display time of each frame ）; Video width and height parameters ; Of each frame of video dts、pts, And convert it to the relative time when the file starts playing （ Unit is ms）; Audio frame dts; Audio sampling frequency 、 The channel number 、 Sampling depth ; A cache is required to store video frames and audio frames and transmit the cache to lower level modules , Each frame cache must carry the pts Time , If you use a fixed cache , Multiple frames or even incomplete frames are stored in the fixed cache at one time , Just send it directly to the decoding module , You need to carry the oldest in each cache pts, And after that, I can pts Perform a dead reckoning restore .
2.1、 When unpacking, you can directly stream the files as required , Read required length , You need to analyze how long you study , Even jump back and forth in the file to read non fixed length data , But this will reduce the burden on your system , Without the support of operating system file system, it will increase IO Number of reads , Increase the time consumption caused by reading files , At this time, you may need to pre cache a fixed length of file data each time when unpacking , Then parse the data in the cache , This is also a complicated logic , Various switching processes are required . In addition, the data frame length cached by the general audio and video playback module is different , You may need a cache of read data frames for both video and audio , They are still a long way from the address currently read in the file .

3、 Both audio and video decoding support frame input and random length input , It is best to input by frame , However, in order to save execution times and make better use of cache resources, you can also enter by cache length , Audio and video decoding requires that the first frame of the input pts And each frame after decoding pts correspond , The decoded data is usually of fixed length , It is also convenient to display and play fixed length data .

4、 The decoded data frames of the audio and video streams are sent to the audio and video synchronization module , The synchronization module sends the audio directly to the playback module for playback , However, it is necessary to record the current playback frame when sending pts、 The length and time of data currently being cached by the audio playback module during playback , Audio playback data is generally obtained through callback or interrupt , At this time, you need to record this information during callback , From this information, the audio time axis is calculated .（ This is the mainstream method of video synchronization to audio , The method of synchronizing audio to video and audio and video to the main time axis is not introduced here ）
4.1、 During video display, it is necessary to record the system time point and frame displayed in the current frame pts, Calculate the video timeline .
4.2、 Through the audio and video synchronization algorithm and the time base of the current audio and video, it is determined that the current video frame is displayed in advance 、 Whether the display is pushed back or the frame is lost . General video leading audio 100ms To fall behind 20ms Human ears can't feel it , Within this range, the video can be displayed at the original frame rate , Otherwise, the current frame will be displayed in advance or later .
4.3、 Generally, the input and output structures of video decoding have private parameters , This parameter allows you to match the input frame order with the output frame order , The private parameter data of the output frame after decoding is the content of the data of the previous frame . be based on I frame 、P frame 、B Frame bidirectional prediction encoding and decoding method , The video decoding sequence is different from the playback sequence , The decoding timestamp is dts, The display timestamp is dts, This mechanism of the video decoding module allows you not to calculate the frame time matching by yourself , Because the input and output of video decoding have frame caching mechanism , It is difficult to calculate the timestamp per frame pts. A video frame before decoding , It may decode several frames , This mechanism also requires you to deal with this situation artificially , Because the decoded video frames have the same private parameters （ That is to say pts Time stamp ）.

5、 The video display module has a fixed vsync The refresh rate , Such as 60Hz, If the frame rate of the video stream is close to the frame rate of the display module, it needs to be set in vsync Refresh the contents of the video memory at the switched time point , Match the timing .

6、 The actual playback speed of the audio playback module may not be the speed of the audio stream standard , Such as 44100Hz, At this time, match the audio time axis .
6.1、 When playing at double speed , To ensure that the audio is not distorted , You also need to resample the audio frame data , Make sure that the frequency in the audio is also adjusted according to the multiple speed , This involves algorithms , Not covered here .