Ffmpeg captures RTSP images for image analysis

Preface

demand ： Use FFmpeg obtain RTSP flow , Grab one of the pictures for image analysis .
gossip ： Originally , My tool is to ARM Used on the machine , Finally, because of the Library , Not used FFmepg Grab the picture . Instead, it adopted Grab the picture , But this tool has a more fatal problem , It's to stop capturing pictures every time , The object of the snapshot must be deleted ( Maybe I didn't find a good way by myself , I hope anyone who knows can give me a guide in the comment area ！), Open every time , Again new, This is a very bad method .
But in this article , I won't use the bad method mentioned above , You can also explain this method when you are free , Maybe I can think of ways to improve .

Text

One 、 Environmental Science

Win11+Qt5.15+MSVC2019+FFmpeg+OpenCV
FFmpeg I forgot the version of , It should be downloaded from the Internet , Download the latest version on the Internet , It should not be a big problem . Download link ：https://ffmpeg.org/releases/?C=N;O=D
OpenCV The version of is the source code I downloaded from the Internet , You can refer to this link to download :Ubuntu 16.04 + Qt 5.11 +opencv 3.4 Perfect configuration （ Personal test , The simplest and perfect way ） And then use VS2019 Compile the Library . If you don't use MSVC2019 It is MinGW, You have to compile it again , Find a tutorial online , It should be quite easy . then , You will compile it again include,bin,lib Put the file in my current location , That's all right. , perhaps , Put yourself in a position , then ,pro It is also possible to modify the file .

Program links ：
File structure :

In the picture above ffmpeg And opencv It's the library file I need to put , Basically, as long as your environment is the same as mine , There should be no need to download , You can open it directly Qt compile , After compiling , Will prompt some dll Library does not , This is the time , Just put the inside ffmpeg and opencv Inside /bin In folder dll Copy the file and put it into your compiled exe In file ( Usually in build Under the table of contents ), such , It should be compiled and passed .

Two 、 Program effect

3、 ... and 、FFmpeg Grab RTSP picture

1. ffmpeg The stop and start of capturing pictures are realized by starting or terminating a thread .

/* Threads */
void VideoPlayer::startPlay()
{
    
 // call  QThread  Of start function   The following... Will be executed automatically run function  run Function is a new thread 
    this->start();
}

void VideoPlayer::stopPlay()
{
    
 // call  QThread  Of start function   The following... Will be executed automatically run function  run Function is a new thread 
// thread()->terminate();
// thread()->wait();
    this->terminate();
    this->wait();
}

2. The core grabs the thread of the picture

/* Threads */
void VideoPlayer::run()
{
    
    /* Defining structure   Call its member function */
    // Input data cache , Number of video and audio streams   Video and audio streaming   file name   Duration   Bit rate   Unpacking, etc 
    AVFormatContext *pFormatCtx;
    AVCodecContext *pCodecCtx;
    AVCodec *pCodec;/* Store decoder information */
    AVFrame *pFrame, *pFrameRGB;/* Store decoder information */
    AVPacket *packet;/* Data packets */
   static uint8_t *out_buffer;

    /* Processing picture pixel data   Image pixel format conversion   Picture stretching, etc  */
    static struct SwsContext *img_convert_ctx;
         /* Video streaming */       /* Images */
    int videoStream, i, numBytes;
      /* decode */   /* Decoding successful */
    int ret, got_picture;

    avformat_network_init();// initialization FFmpeg Network module 
    av_register_all();// initialization FFMPEG  Only by calling this can the encoder and decoder work properly ( Discard function )


    pFormatCtx = avformat_alloc_context();// Initialize memory 

    //AVDictionary yes FFmpeg Key value pair storage tool ,FFmpeg Regular use AVDictionary Set up / Read internal parameters 
    AVDictionary *avdic=NULL;
    char option_key[]="rtsp_transport";
    char  m_bTcp;
    av_dict_set(&avdic,option_key,m_bTcp ? "udp" : "tcp",0);
    char option_key2[]="stimeout";
    char option_value2[]="3000000";
    av_dict_set(&avdic, "buffer_size", "1024000", 0);                 // Image quality optimization 
    av_dict_set(&avdic,option_key2,option_value2,0);
    //char url[]="rtsp://admin:[email protected]/stream0";/* Webcam data */
    QByteArray ba=m_sUrlAddress.toLocal8Bit();
    char* url = ba.data();
    /*avformat_open_input function */
    // Parameter one ： Point to user provided AVFormatContext（ from avformat_alloc_context Distribute ） The pointer to .
    // Parameter two ： To open the flow of url
    // Parameter 3 ：fmt If it is not empty , This parameter forces the use of a specific input format . otherwise 6 The format will be detected automatically .
    // Parameter 4 ： contain AVFormatContext and demuxer A dictionary of private options . return , This parameter will be destroyed and replaced with options that cannot be found 
    if (avformat_open_input(&pFormatCtx, url, NULL, &avdic) != 0)    // Open multimedia and get information 
    {
    
        printf("can't open the file. \n");
        return;
    }
    if(avdic != NULL)
    {
    
        av_dict_free(&avdic);
    }
    qDebug()<<"--->z pFormatCtx->streams[videoStream]->codec"<<pFormatCtx->streams[0]->codec->codec_id;
    // Get video stream information 
    /*avformat_find_stream_info function */
    // Parameter one ： Media file context .
    // Parameter two ： Dictionaries , Some configuration options . /* Media handle */
    if (avformat_find_stream_info(pFormatCtx, NULL) < 0)
    {
    
        printf("Could't find stream infomation.\n");
        return;
    }
    videoStream = -1;/* No video streaming */
    // Loop to find the stream information contained in the video , Until you find a video type stream 
    /* pFormatCtx function */
    //unsigned int nb_streams  Current number of streams 
    //AVStream **streams;  Pointer array   Video stream and voice stream */
    for (i = 0; i < pFormatCtx->nb_streams; i++)
    {
    
        qDebug()<<"--->z run1"<<pFormatCtx->streams[i]->codec->codec_type;
        if (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)//codec Discard function 
        {
    
            videoStream = i;
        }
    }
    // If videoStream by -1  The video stream was not found 
    if (videoStream == -1)
    {
    
        printf("Didn't find a video stream.\n");
        return;
    }

    // Print stream information 
    // Be careful ： Fill in the last parameter 0, Print input stream ; Fill in the last parameter 1, Printout stream 
    av_dump_format(pFormatCtx, 0, url,0);

    // Find the decoder , Get a pointer to the codec context of the video stream 
    pCodecCtx = pFormatCtx->streams[videoStream]->codec;
    // After unpacking, start from avstream Get in the structure CodecID（ Specify the format stream ）
    pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
    qDebug()<<"--->z CVlcCameraCapture::run()2"<<pCodec<<pCodecCtx->codec_id<<pCodec<<videoStream;
    // Set encoder parameters ( The influence of different parameters on the quality or size of video editing )
    pCodecCtx->bit_rate =0;   // Initialize to 0  Bit rate 
    pCodecCtx->time_base.num=1;  // The next two lines ： A second 25 frame 
    pCodecCtx->time_base.den=25;
    pCodecCtx->frame_number=1;   // One video frame per packet 

    /* If encoder equals NULL  Encoder not found */
    if (pCodec == NULL)
    {
    
        printf("Codec not found.\n");
        return;
    }

    // Turn on the decoder 
    if (avcodec_open2(pCodecCtx, pCodec, NULL) < 0)
    {
    
        printf("Could not open codec.\n");
        return;
    }

    pFrame = av_frame_alloc();    // establish   Store decoder information */
    pFrameRGB = av_frame_alloc(); // establish   Store decoder information */

                                  // After decoding h264 Data to RGB32
    img_convert_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height,
                 pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height,
                        AV_PIX_FMT_RGB32, SWS_BICUBIC, NULL, NULL, NULL);


    // The pixel format of the image   The pixel width of the image   The pixel height of the image ( Calculate the picture in this format , How many bytes are needed to store )
    numBytes = avpicture_get_size(AV_PIX_FMT_RGB32, pCodecCtx->width,pCodecCtx->height);//( Discard function )
    qDebug() << numBytes;        // How many bytes are needed to store 

     out_buffer = (uint8_t *) av_malloc(numBytes * sizeof(uint8_t));
     /* Divide up the allocated space */
     // Divide up what was allocated in the previous step buffer.
     av_image_fill_arrays(pFrameRGB->data, pFrameRGB->linesize, out_buffer, AV_PIX_FMT_RGB32, pCodecCtx->width, pCodecCtx->height, 1);


    int y_size = pCodecCtx->width * pCodecCtx->height;
    packet = (AVPacket *) malloc(sizeof(AVPacket)); // Apply for the size of a video frame packet 
    av_new_packet(packet, y_size); // Distribute packet The data of , by packet Allocate a specified size of memory 

     int as = 0;
    while (1)
    {
    
        //av_read_frame
        // Returns the next frame of the stream . This function returns the contents stored in the file , Do not validate valid frames . Get the frame stored in the file ,
        // And return a for each call . It won't omit invalid data between valid frames , In order to give the decoder the maximum information that can be used to decode .
        // return 0 To be successful , Less than 0 It's a mistake , Greater than 0 At the end of the file , So greater than or equal to 0 It's a return to success 
        // Every decoded video frame , You need to call  av_read_frame() Get the compressed data of a frame of video , Then the data can be decoded 
        if (av_read_frame(pFormatCtx, packet) <  0)
        {
    
           qDebug("a == %d\n",++as);

           if(as == 4)
           {
    
           qDebug("  The connection ended abnormally \n");
          thread()->terminate();
          thread()->wait();
          this->terminate();
          this->wait();
           }
           continue;
        }

          if(as != 0)
          {
    
              as = 0;
          }
        if (packet->stream_index == videoStream)
        {
    
           ret = avcodec_send_packet(pCodecCtx, packet);                    // Send data to ffmepg, Put it in the decoding queue 
           got_picture = avcodec_receive_frame(pCodecCtx, pFrame);          // Take the successful decoding queue 1 individual frame

            if (ret < 0)
            {
    
                usleep(1000);
                printf("decode error.\n");
                continue;
            }

            if (!got_picture)
            {
    
                 // Color space conversion , Last output to out_buffer
                sws_scale(img_convert_ctx,(uint8_t const * const *) pFrame->data,
                        pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data,
                        pFrameRGB->linesize);//sws_scale Libraries can be implemented simultaneously in one function ：1. Image color space conversion ;2. Resolution scaling ;3. Before and after image filtering processing .

                // Put this RGB data   use QImage load 
                QImage tmpImg((uchar *)out_buffer,pCodecCtx->width,pCodecCtx->height,QImage::Format_RGB32);
                QImage image = tmpImg.copy(); // Make a copy of the image   Pass to the interface display 
                emit sig_GetOneFrame(image);  // Sending signal 
             }
         }

        // Release a package .
        av_free_packet(packet); // Release resources , Otherwise, the memory will keep rising ( Discard function )
        av_packet_unref(packet);
        memset(out_buffer,0,sizeof(out_buffer));

    }
    av_free(out_buffer);
    av_free(pFrameRGB);
    avcodec_close(pCodecCtx);// Turn off the given avcodeContext And release all the data associated with it 
    if(NULL != pCodecCtx){
    
        avcodec_free_context(&pCodecCtx);
        avdic = NULL;
    }
    if(NULL != pFormatCtx){
    
    avformat_close_input(&pFormatCtx);// Turn off open input pFormatCtx. Release it and all its contents and set it to empty .
    pFormatCtx = NULL;
    }

}

The notes above should be clearly written , If it's not clear , You can see the following picture .

Basically, this process is relatively elementary , It is worth remembering the following .

Four 、 After image capture , Image analysis

Reference resources

FFmpeg

1. FFmpeg Examples of decoding audio and video and records of problems encountered （ Two ）
2. The sea, 、 Dahua webcam RTSP URL Format composition and parameter configuration
3. ffmpeg The decoder uses
4. ffmpeg Download link
5. Linux Cross compile under FFMPEG And X264 library ： Target board friendly arm Tiny4412 Development board _EXYNOS4412（ARMV7_32 position ）
6. ubuntu Cross compile under X264 and FFMPEG To RK3399 platform ( compiler :aarch64-linux-gcc)

OpenCV

1. OpenCV + CPP series （ Six ） Target detection and counting
2. [ Commonly used tools ] OpenCV_contrib Library in windows Compile and use the guide
3. opencv Contour based template matching
4. Ubuntu 16.04 + Qt 5.11 +opencv 3.4 Perfect configuration （ Personal test , The simplest and perfect way ）
5. OpenCV Development ：ubuntu18.04 Cross compile under OpenCV3.4.9 To ARM64 Bit platform RK3399(aarch64-linux-)
6. opencv Edge detection of learning