Sorting out the soft and hard decoding methods of ffmpeg

background

Project use QT Develop monitoring software , Integrated Haikang 、 Depending Yu 、 Dahua 、 Hua mai 、 And network streaming devices . Brand equipment use SDK control , The network streaming device is used by itself FFmpeg Library to solve . If the network stream device decodes and multiplexes at the same time , There will be CPU High occupancy 、 Operation of the Caton . If there are more video streams, such as 16 road , Or decode more complex ones such as H265, that CPU Will be unable to support , Even an abnormal crash is possible .

Problem analysis

Video playback requires a certain amount of computing power . It is mainly reflected in decoding and rendering , Default FFmpeg Is software decoding （ Soft solution ）, Then, the decoded data is converted to the next format , Re pass QPainter The rendering is displayed , Is to use CPU To calculate . Therefore, when the number of paths increases ,CPU The use of resources is increasing , There will be a phenomenon of insufficient resources .
But this phenomenon is normal , A lot of work ,CPU The more resources will be consumed , So all we have to do is put CPU To share the task with GPU, Hardware decoding （ Hard solution ）.

Hardware decoding

1. Ideas

GPU, Graphics processor , It can be simply considered as various nuclear manifestations , Independent graphics card .CPU an able man is always busy , But there are limits ,GPU Generally, it is specialized in image related processing . Then you can hand over the decoding and rendering of audio and video to GPU To do it , Will greatly liberate CPU, Of course it's different GPU It is also different in capacity , therefore GPU There is no guarantee that the decoding effect will be better than CPU good （ Not too bad ）, It mainly replaces CPU Share the work of decoding .

2. The way

How to use hard solutions ？ use Baidu Search , You will see a few words that often appear ：DXVA2、 D3D11VA、QSV、libmfx、CUDA、VAAPI etc. . It's necessary to know DirectX（Windows Multimedia library interface provided by the system ）,PC Hardware coding and decoding of platform video 、 Rendering and so on . Generally speaking, there are many hard solutions , The details are not introduced here , Let's understand... From the perspective of use , For details, please refer to FFmpeg Hard solution of the sample program , These methods can be divided into two categories ： One is provided by the graphics card itself API; One is system encapsulated API,.

1. Provided by the graphics card itself , It can be understood as the instruction manual of a circuit board , You can operate directly with the corresponding interface GPU Resources to do the desired processing , Such as VAAPI、libmfx etc.
2. System encapsulated , System level API Interface , Ignore hardware differences , If only you use Windows System , You can use it DXVA2 And other interfaces provided by the system , It doesn't matter how the system operates GPU Of

Hardware rendering

Decode by hardware , It has been able to greatly reduce CPU The burden of . The principle is CPU Provide the original audio and video data to GPU,GPU To do the processing and get the decoded data ,CPU Do not participate in the decoding process . There is a problem here , How to deal with the decoded data ？

1. The decoded data is from GPU Then transmit it to CPU, call QPainter use CPU To render the image
2. Self marketing ,GPU Decoded data in , Directly through GPU Render image in , Such as QOpenGLWidget

Both methods can be selected according to the actual situation , But obviously , The second way is more liberating CPU. Because the main decoding of the whole playback process 、 The rendering work is left to CPU Did , here CPU Even if there is still some processing occupation , It is also very low .

test

Through the above two steps , After actual testing , At the same time processing 16 Three way video stream （H265、H264 code ）,CPU There is a significant decline , Normal operation without jamming , Play simultaneously on another host with better performance 25 The road can also be played smoothly . Compared with soft solution , Great performance improvement , You can view it in the task manager CPU、GPU The occupancy of .
thus , Problem solved . The program complexity increases slightly , If you only play a single stream or do not occupy too much CPU resources , The default soft solution is OK .

explain

This is an explanation FFmpeg soft 、 Hard decoding the main idea of the article , Find out why , How to do it , You can search the relevant articles for details , There are a lot of articles by big men . As long as we grasp the trunk and clarify our thinking , The rest goes step by step , Naturally .