Why is the frame rate calculated by opencv wrong?

▲ Scan the QR code in the picture to learn more about the audio and video technology conference ▲

---

author ： Wang Wei

edit ：Alex

lead   said

We have a platform to periodically detect the live stream data on the Internet , For example, black / White screen detection 、 Static picture detection …… In the process of testing , We will estimate the number of frames to be calculated according to the frame rate of the extracted live stream , for example , If you want to test 5s Live streaming of , The frame rate of the live stream is 20fps, The number of frames to be calculated is 100. Then one day , We found that , The platform began to time out in a large area , Before that, we just needed 2s Can complete the calculation , Now we need to 30+ minute . After checking , We found that , The reason for calculating the timeout is because OpenCV The calculated frame rate is 2000, As a result, the number of frames to be calculated is reduced from 100 Change into 10000, This in turn causes a computation timeout .

1、OpenCV How to calculate the frame rate

For a detailed description of this problem, see OpenCV Issues 21006[1]. The simulated live stream segment of this problem test.ts You can click the link to download ：

https://pan.baidu.com/share/init?surl=RY0Zk5C_DOEwTXYe2SLFEg, The download extraction code is x87m.

If you use the following code to get test.ts Of fps,

const double FPS = cap.get(cv::CAP_PROP_FPS);
std::cout << "fps: " << FPS << std::endl;

You can get ：

$ fps: 2000

use ffprobe Analyze the video , You can get ：

codec_name=h264
r_frame_rate=30/1
avg_frame_rate=0/0
……

from opencv/modules/videoio/src/cap_ffmpeg_impl.hpp[2] in , We found that fps from CvCapture_FFMPEG::get Come by calculation , The calculation logic is as follows ：

double fps = r2d(ic->streams[video_stream]->avg_frame_rate);
if (fps < eps_zero) {
    fps = 1.0 / r2d(ic->streams[video_stream]->codec->time_base);
}

2、 Why? OpenCV The frame rate is wrong

utilize test_time_base.cpp[3], We can get ：

time_base: 1/2000
framerate: 0/0
avg_framerate: 0/0
r2d(ic->streams[video_stream]->avg_frame_rate) = 0

therefore OpenCV Adopted ：

1.0 / r2d(ic->streams[video_stream]->codec->time_base)

To calculate the fps. And here time_base = 1/2000, therefore , resulting fps yes 2000.

in other words ,AVStream->codec->time_base The value of results in OpenCV Get one that looks wrong fps. that ,AVStream->codec->time_base Why is this value ？FFmpeg How to calculate this field ？

3、FFmpeg How to calculate

AVCodecContext.time_base

AVStream->codec->time_base yes AVCodecContext As defined in time_base Field , according to libavcodec/avcodec.h[4] As defined in , For decoding ,time_base Has been abandoned , Need to use framerate To replace time_base. also , For a fixed frame rate ,time_base = 1/framerate, But not always .

utilize H264Naked[5] Yes test.ts Corresponding H.264 Code stream analysis , We get SPS.Vui Information ：

timing_info_present_flag :1
num_units_in_tick :1
time_scale :2000
fixed_frame_rate_flag :0

You can see from it ,test.ts Is a non fixed frame rate video . from test_time_base.cpp[3] Look at the result ,test.ts In the video ,framerate = 0/0, and time_base = 1/2000.

Don't , For non fixed frame rate video ,time_base and framerate There's no connection ？ If there is a connection , What kind of operation can produce such a result ？ This time_base How exactly is it calculated ？ What is the relationship between framerate It doesn't matter ？ A series of problems followed ……

The source code in front , No secret . Next , With this question , Let's analyze FFmpeg How to deal with time_base Of .

3.1  avformat_find_stream_info

stay FFmpeg in ,avformat_find_stream_info() Yes ic->streams[video_stream]->codec To initialize , So we can go from avformat_find_stream_info() To analyze .

from libavformat/avformat.h[6] in , It can be learned that avformat_open_input() Will open the video stream , Read relevant information from , Then stored in AVFormatContext in , But sometimes , The information obtained here is not complete , So you need to call avformat_find_stream_info() To get more information .

It should be noted that ：

avformat_find_stream_info() Will try to decode some video frames to obtain the required information .

/** 
* Read packets of a media file to get stream information. This 
* is useful for file formats with no headers such as MPEG. This 
* function also computes the real framerate in case of MPEG-2 repeat 
* frame mode. 
* The logical file position is not changed by this function; 
* examined packets may be buffered for later processing. 
* 
* @param ic media file handle 
* @param options  If non-NULL, an ic.nb_streams long array of pointers to
*                 dictionaries, where i-th member contains options for 
*                 codec corresponding to i-th stream. 
*                 On return each dictionary will be filled with options that
*                 were not found. 
* @return >=0 if OK, AVERROR_xxx on error 
* 
* @note this function isn't guaranteed to open all the codecs, so 
*       options being non-empty at return is a perfectly normal behavior. 
* 
* @todo Let the user decide somehow what information is needed so that 
*       we do not waste time getting stuff the user does not need. 
*/
int avformat_find_stream_info(AVFormatContext*ic, AVDictionary **options);

avformat_find_stream_info() The overall logic of is roughly as shown in the figure below , Special attention should be paid to the 7 A step ：

3.2  avformat_find_stream_info() Description of the important steps of

STEP 1   Set the number of threads , avoid H.264 Multithreaded decoding does not put SPS/PPS Information is extracted to extradata.

STEP 2   Set up AVStream *st,st It will be transmitted to... In subsequent function calls try_decode_frame().

STEP 3 Relatively simple , No more details here .

STEP 4 Set up AVCodecContext *avctx Transparent st->internal->avctx, In subsequent decoding function calls , This is what has been transmitted all the time avctx, therefore , The execution process starts here ,FFmpeg All used are st->internal->avctx, instead of st->codec, Pay special attention here . The number of decoding threads will also be set here , Its purpose and STEP 1 It's consistent .

STEP 5 Because the number of decoding threads was set to 1, So here we call

ret = avctx->codec->decode(avctx, frame, &got_frame, pkt)

To decode and calculate avctx->framerate. Be careful , Here avctx In fact, it came from the outside st->internal->avctx. Calculation framerate The logic of how to calculate framerate Part introduction .

STEP 6 According to the decoder framerate Information to calculate avctx->time_base, Notice that this is actually st->internal->avctx->time_base. according to   How to calculate framerate You know , here framerate = {1000, 1}. according to AVCodecContext.ticks_per_frame The introduction of ,ticks_per_frame = 2. therefore , here avctx->time_base = {1, 2000}：

avctx->time_base = av_inv_q(av_mul_q({1000, 1}, {2, 1})) = {1, 2000}

STEP 7 This step can be described as “ Practise deception , Build the plank road to hide the storehouse ”. This step is to solve API Forward compatibility of , Made a replacement , hold st->internal->avctx->time_base Assigned to st->codec->time_base, But the s****t->avg_frame_rate Assigned to st->codec->framerate. therefore ：

st->codec->time_base = {1, 2000}
st->codec->framerate = {0, 0}

st->codec->time_base And st->codec->framerate There is no relationship between , But and st->internal->avctx->framerate of . Its essence , Is and sps.time_scale,sps.num_units_in_tick of .

st->internal->avctx->time_base.num =
sps->num_units_in_tick * 
st->internal->avctx->ticks_per_frame

st->internal->avctx->time_base.den = sps->time_scale * 
   st->internal->avctx->ticks_per_frame;

st->internal->avctx->time_base = {sps->num_units_in_tick,
sps->time_scale}

3.3  internal->avctx->time_base & internal->framerate

So actually ,internal->avctx->time_base by ：

avctx->time_base = sps->num_units_in_tick /
sps->time_scale

and internal->avctx->framerate It is ：

avctx->framerate = sps->time_scale /
(sps->num_units_in_tick * avctx->ticks_per_frame)

therefore , about H.264 In terms of code stream ,time_base = 1 / (2 * framerate), instead of 1 / framerate.

That's why

libavcodec/avcodec.h[4] In the said ：

* This often, but not always is the inverse of the frame rate or field rate
* for video.

From the above analysis, we can know ：

avctx->framerate = 1 / (avctx->time_base * avctx->ticks_per_frame)

therefore , When st->avg_frame_rate = 0 when ,OpenCV Calculation fps The logic of is wrong .

stay H.265 in ,ticks_per_frame = 1, So for H.265 The coding ,OpenCV There is no such problem . have access to Zond 265 [7] Tools to analyze a H.265 Video stream , Then compare it with OpenCV as well as FFmpeg The results are verified .

meanwhile , As shown above STEP 7 The grafting of trees in the results in test_time_base.cpp[3] Result ：

st->codec->framerate: 0/0
st->codec->time_base: 1/2000

3.4 ff_h264_decoder

libavcodec/decode.c [8] Medium

decode_simple_internal() The corresponding decoder will be called to decode （STEP 5）. And as shown before ,test.ts by H.264 Encoded video stream , So... Will be called here H.264 Decoder to decode . stay FFmpeg in ,H.264 The decoder is located at libavcodec/h264dec.c[9] As defined in

const AVCodec ff_h264_decoder.

const AVCodec ff_h264_decoder = {
    .name                  = "h264",
    .type                  = AVMEDIA_TYPE_VIDEO,
    .id                    = AV_CODEC_ID_H264,
    .priv_data_size        = sizeof(H264Context),
    .init                  = h264_decode_init,
    .close                 = h264_decode_end,
    .decode                = h264_decode_frame,
    ......
};

In the figure above STEP 5 in ,

ret = avctx->codec->decode(avctx, frame, &got_frame, pkt);

The actual call is ：

ff_h264_decoder->h264_decode_frame(avctx, frame, &got_frame, pkt);

And here avctx That is to say

try_decode_frame() It's passed down from Zhongtou st->internal->avctx, That is... In the figure above STEP 4.

3.5  h264_decode_frame

h264_decode_frame() The overall logic of is shown in the figure below ：

3.6  AVCodecContext.ticks_per_frame

We'll use that later ticks_per_frame To calculate framerate. stay STEP 6 Middle computation time_base This value is also used when . therefore , It is necessary to make a special explanation . stay H.264 In decoder ,ticks_per_frame=2, The specific value can be seen from the following ：

• libavcodec/avcodec.h [4] Field description in ：

/**
* For some codecs, the time base is closer to the field rate than the frame rate. 
* Most notably, H.264 and MPEG-2 specify time_base as half of frame duration 
* if no telecine is used ... 
* 
* Set to time_base ticks per frame. Default 1, e.g., H.264/MPEG-2 set it to 2.
 */
 int ticks_per_frame;

• libavcodec/h264dec.c [9] Medium  h264_decode_init()：

avctx->ticks_per_frame = 2;

4、 How to calculate framerate

STEP 1 According to the overall calculation process , Here h It's actually

avformat_find_stream_info() Medium

st->internal->avctx->priv_data.h It will be transmitted to all subsequent processes , This must be noted .

STEP 2 Here you will first get sps Information about , For subsequent calculation , We can take another look at test.ts sps[10] Information about .

timing_info_present_flag :1
num_units_in_tick :1
time_scale :2000
fixed_frame_rate_flag :0

STEP 3 according to sps Calculation of relevant information framerate, Above STEP 6 Middle computation time_base Use of framerate It is calculated here . because timing_info_present_flag = 1, Therefore, the calculation will be performed framerate The logic of ：

avctx->framerate.den = sps->num_units_in_tick * h->avctx->ticks_per_frame = 1 * 2 = 2
avctx->framerate.num = sps->time_scale = 2000
avctx->framerate = (AVRational){1000, 1}

therefore ,

st->internal->avctx->framerate = {1000, 1}

however , because avctx->time_base={1,2000}, therefore OpenCV The calculated frame rate is 2000. The reason for this inconsistency is ,OpenCV In the use of codec->time_base The frame rate is calculated without considering ticks_per_frame. therefore , about OpenCV for , The correct way to calculate the frame rate should be ：

double fps = r2d(ic->streams[video_stream]->avg_frame_rate);
if (fps < eps_zero) {
fps = 1.0 / r2d(ic->streams[video_stream]->codec->time_base * ic->streams[video_stream]->codec->ticks_per_frame);
}

junction   On

Through the above analysis, we can know ：

• FFmpeg In the calculation AVCodecContex Medium framerate and time_base When , use ：

  o sps.time_scale

  o sps.num_units_in_tick

  o AVCodecContex.ticks_per_frame

• stay FFmpeg in ,framerate and time_base The relationship is ：

  o framerate = 1 / (time_base * ticks_per_frame)

  o time_base = 1 / (framerate * ticks_per_frame)

• For non H.264/MPEG-2,

  ticks_per_frame=1, therefore framerate and time_base It is reciprocal relationship . And for H.264/MPEG-2 for ,ticks_per_frame=2, therefore , At this time, the two are not reciprocal relations . thus ,FFmpeg Just say ,framerate and time_base It is usually a reciprocal relationship , But not always .

• stay OpenCV in , about H.264/MPEG-2 In terms of video , When

  AVStream.avg_frame_rate=0 when , Its calculation fps The logic of existence BUG.

• Because in decoding ,

  AVCodecContex.time_base Has been abandoned , meanwhile AVStream.avctx It's abandoned , and

  avformat_find_stream_info() In order to be compatible with the old API, So I will take advantage of AVStream.internal.avctx And other information AVStream.avctx. and AVStream.avctx.time_base Taken from the AVStream.internal.avctx,AVStream.avctx.framerate From AVStream.framerate.

notes ：

[1] https://github.com/opencv/opencv/issues/21006

[2] https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_FFmpeg_impl.hpp

[3] https://github.com/wangwei1237/wangwei1237.github.io_src/blob/master/source/_posts/Why-OpenCV-Get-the-Wrong-FPS/test_time_base.cpp

[4]https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/avcodec.h

[5] https://github.com/shi-yan/H264Naked

[6] https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/avformat.h

[7] https://www.dektec.com/products/applications/Zond/

[8]https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/decode.c

[9]https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/h264dec.c

[10] https://wangwei1237.github.io/2021/11/26/Why-OpenCV-Get-the-Wrong-FPS/#sps

Author's brief introduction ：

Wang Wei ,17 Brother , Baidu Senior Test Engineer . Technical director of Baidu video quality evaluation , In solving the standardization of video quality evaluation 、 I have rich practical experience in confidence , Built Baidu's first systematic video quality evaluation service platform , And serve multiple video services .

---