GStreamer ffmpeg avdec decoded data flow analysis

avdec_xxx yes gst-libav Decoding plug-in in , The corresponding file is gstavviddec.c.

avdec When initializing ,decoder The following functions are initialized in , therefore libav Of avviddec Plug in , Main data processing , Just pay attention to the implementation of several functions ：

  viddec_class->handle_frame = gst_ffmpegviddec_handle_frame;
  viddec_class->start = gst_ffmpegviddec_start;
  viddec_class->stop = gst_ffmpegviddec_stop;
  viddec_class->flush = gst_ffmpegviddec_flush;
  viddec_class->finish = gst_ffmpegviddec_finish;

viddec_class->handle_frame receive upstream Data processing ,viddec_class->finish Is in the base class sink_event in , received EOS It will be called later .

ffmpeg Call in @gst_video_decoder_finish_frame Data to be decoded push To downstream, After the following process ：

gst_video_decoder_decode_frame： Base class decoder frame
- gst_ffmpegviddec_handle_frame
  - gst_ffmpegviddec_frame
    - gst_ffmpegviddec_video_frame
      - gst_video_decoder_finish_frame

gst_ffmpegviddec_handle_frame function

• take frame->input_buffer Copied to the ffmpeg Of packet in
• avcodec_send_packet The above packet Send it to the decoder
• from gst_ffmpegviddec_frame Get decoding buffer And send it to downstream

gst_ffmpegviddec_video_frame function

gst_ffmpegviddec_video_frame The work done in is listed as follows , Last gst_video_decoder_finish_frame Is the most critical one , Will decode the buffer Continue to combine GstBuffer, issue downstream To deal with .

gst_ffmpegviddec_video_frame
- gst_ffmpegviddec_do_qos
- avcodec_receive_frame
- gst_ffmpegviddec_negotiate
  - update_video_context
  - gst_ffmpeg_pixfmt_to_videoformat
  - gst_video_decoder_set_output_state
  - gst_video_decoder_negotiate
- gst_video_decoder_finish_frame( The last crucial step ,push To downstream )

After decoding QOS

From the stack order above , After decoding, you need to do QOS, Yes AVDiscard In the definition of ,AVDiscard The types of are defined as follows , This is in gst_ffmpegviddec_do_qos You can see in the function , The decoder executes before decoding the next frame qos Calculation , Set up skip_frame sign , If the situation is very bad , Then skip to the next key frame .

enum AVDiscard{
    
    /* We leave some space between them for extensions (drop some * keyframes for intra-only or drop just some bidir frames). */
    AVDISCARD_NONE    =-16, ///< discard nothing  Nothing is discarded 
    AVDISCARD_DEFAULT =  0, ///< discard useless packets like 0 size packets in avi  Discard useless packets 
    AVDISCARD_NONREF  =  8, ///< discard all non reference  Discard all non reference frames 
    AVDISCARD_BIDIR   = 16, ///< discard all bidirectional frames  Discard all bidirectional frames 
    AVDISCARD_NONINTRA= 24, ///< discard all non intra frames  Discard all non intra frames 
    AVDISCARD_NONKEY  = 32, ///< discard all frames except keyframes  Discard all frames except keyframes 
    AVDISCARD_ALL     = 48, ///< discard all  Discard it all 
};

gst_ffmpegviddec_negotiate

gst_ffmpegviddec_negotiate Lieutenant general avcodec Of AVCodecContext The relevant information in is converted into gstreamer Medium GstVideoInfo, And then call gst_video_decoder_negotiate Progress and downstream Negotiation of .

This is to obtain caps Method of information ：

  in_s = gst_caps_get_structure (ffmpegdec->input_state->caps, 0);
  if (!gst_structure_has_field (in_s, "interlace-mode")) {
    
      // ...
  }
  // ...

gst_video_decoder_negotiate

Negotiate the current configuration with downstream elements GstVideoCodecState, Cancel in any case flag GST_PAD_FLAG_NEED_RECONFIGURE. But if the negotiation fails , You need to mark .

gst_ffmpegviddec_get_buffer2

stay avviddec in ,get_buffer2 yes ffmpeg The decoded data is converted into gstreamer Medium GstVideoCodecFrame type ,Fill avpicture The process of , Will be GstVideoCodecFrame Corresponding buffer Address translation to avpicture Inside , I use picture Will write data into it when .

  // picture->reordered_opaque : system_frame_nunber;
  frame = gst_video_decoder_get_frame (GST_VIDEO_DECODER (ffmpegdec), picture->reordered_opaque);
  if (G_UNLIKELY (frame->output_buffer != NULL))
    goto duplicate_frame;

  if (picture->opaque) {
    
    dframe = picture->opaque;
    dframe->frame = frame;
  } else {
    
    picture->opaque = dframe =
        gst_ffmpegviddec_video_frame_new (ffmpegdec, frame);
  }


 /* Fill avpicture */
  if (!gst_video_frame_map (&dframe->vframe, &ffmpegdec->pool_info,
          dframe->buffer, GST_MAP_READWRITE))
    goto map_failed;
  dframe->mapped = TRUE;

  for (c = 0; c < AV_NUM_DATA_POINTERS; c++) {
    
    if (c < GST_VIDEO_INFO_N_PLANES (&ffmpegdec->pool_info)) {
    
      picture->data[c] = GST_VIDEO_FRAME_PLANE_DATA (&dframe->vframe, c);
      picture->linesize[c] = GST_VIDEO_FRAME_PLANE_STRIDE (&dframe->vframe, c);

      if (ffmpegdec->stride[c] == -1)
        ffmpegdec->stride[c] = picture->linesize[c];

      /* libav does not allow stride changes, decide allocation should check * before replacing the internal pool with a downstream pool. * https://bugzilla.gnome.org/show_bug.cgi?id=704769 * https://bugzilla.libav.org/show_bug.cgi?id=556 */
      g_assert (picture->linesize[c] == ffmpegdec->stride[c]);
    } else {
    
      picture->data[c] = NULL;
      picture->linesize[c] = 0;
    }
    GST_LOG_OBJECT (ffmpegdec, "linesize %d, data %p", picture->linesize[c],
        picture->data[c]);
  }

gst_ffmpegviddec_finish

gst_ffmpegviddec_finish At the end GST_EVENT_EOS Called after the event , Next will call avcodec_flush_buffers data-driven flush operation .

gst_video_decoder_sink_event_default
    - gst_ffmpegviddec_finish
    	- gst_ffmpegviddec_drain(drain: Flow away )
      		- gst_ffmpegviddec_frame
       	 		- gst_ffmpegviddec_video_frame