Introduction to the browser video frame operation method requestvideoframecallback()

Recently, I found a cool application scenario , Act on the camera , Then the analyzer animation is rendered to  3D  Model , After understanding its implementation principle , Take some study notes for some technical points . 

requestVideoFrameCallback()

  Is one of the technical points , Using this method can efficiently process video in the browser .

HTMLVideoElement.requestVideoFrameCallback()

  It's a new one WEB API,2021  year  1  month  25  Draft submitted on .

requestVideoFrameCallback()

  Method allows WEB Developers register a callback method , The callback method runs in the rendering step when a new video frame is sent to the compositor . This is to allow developers to perform efficient video operations per frame , For example, video processing and painting on canvas （ screenshots ）、 Video analysis or synchronization with external audio sources .

And  requestAnimationFrame()  The difference between

Can pass API Execution method  

drawImage()

  Draw the video frame to the canvas and other operations will be synchronized with the frame rate of the video played on the screen as much as possible . And usually trigger about every second  

60

  Time of  

window.requestAnimationFrame()

  Different ,

requestVideoFrameCallback()

  Bound with the actual video frame rate , But there is also an important exception ：

The effective rate of running callback is the smaller rate between video rate and browser rate . This means that in  

60Hz

  Drawn in the browser  

25fps

  The video will be  

25Hz

  Trigger callback . In the same  

60Hz

  In the browser  

120fps

  The video will be in  

60Hz

  Trigger callback .

Because of it and  

window.requestAnimationFrame()

  be similar , This method was originally proposed as  

video.requestAnimationFrame()

, Finally use the new name  

requestVideoFrameCallback()

, This was agreed after a long discussion , This name is relatively more intuitive .

In front end development , Usually in the use of some new WEB API Its availability needs to be tested when , You can also use the following code to detect ：

if ("requestVideoFrameCallback" in HTMLVideoElement.prototype) {
 //  Support the corresponding API
}


Browser support

Sure

Click the link to check the corresponding support

.

Usage method

If ever used  

requestAnimationFrame()

  Method , Then it will be right immediately  

requestVideoFrameCallback()

  The method is no stranger . Register an initial callback , Then re register when the callback triggers .

const videoFrameCallback = (now, metadata) => {
 console.log(now, metadata);
 //  Re register the callback to get notification about the next frame .
 videoElement.requestVideoFrameCallback(videoFrameCallback);
};
//  Initially register callback , In order to be notified at the first frame .
videoElement.requestVideoFrameCallback(videoFrameCallback);


In the callback ,

now

  It's a  

DOMHighResTimeStamp

,

metadata

  It's a  

VideoFrameMetadata

  Dictionaries , Has the following properties ：

• presentationTime
  ：
  DOMHighResTimeStamp
    type , The time when the user agent submits the frame for synthesis .

• expectedDisplayTime
  ：
  DOMHighResTimeStamp
    type , The time the user agent expects the framework to be visible .

• width
  ：
  unsigned long
    type , The width of the video frame , In pixels .

• height
  ：
  unsigned long
    type , The height of the video frame , In pixels .

• mediaTime
  ：
  double
    type , Media presentation timestamp  (PTS), In seconds of rendered frames （ for example , It's in  
  video.currentTime
    Timestamp on the timeline ）.

• presentedFrames
  ：
  unsigned long
    type , Submit a count of the number of frames used for synthesis . Allow the client to determine  
  VideoFrameRequestCallback
    Whether frames are lost between instances .

• processingDuration
  ：
  double
    type , The slave will have the same render timestamp as this frame  (PTS)  Encoded packet （ for example , And  
  mediaTime
    identical ） The duration of submitting to the decoder until the decoded frame is ready for rendering （ In seconds ）.

about  

WebRTC

  Applications , Other properties may appear ：

• captureTime
  ：
  DOMHighResTimeStamp
    type , For video frames from local or remote sources , This is the time when the camera captures the frame . For remote sources , Use clock synchronization and  
  RTCP
    The sender reports to estimate the capture time , In order to  RTP  Timestamp is converted to capture time .

• receiveTime
  ：
  DOMHighResTimeStamp
    type , For video frames from remote sources , This is the time when the platform receives the encoded frame , That is, the time when the last packet belonging to the frame is received through the network .

• rtpTimestamp
  ：
  unsigned long
    type , Associated with this video frame  RTP  Time stamp .

Please note that , In some cases ,

width

  and  

height

  Possible and  

videoWidth

  and  

videoHeight

  Different （ for example , Deformed video may have rectangular pixels ）.

In the parameter list above, it is particularly noteworthy that  

mediaTime

 , stay  

Chromium

  In the implementation of , Use audio clock as support  

video.currentTime

  The source of time . and  

mediaTime

  Directly by the frame  

presentationTimestamp

  fill . If you want to accurately identify frames in a reproducible manner , Including accurately identifying missed frames , So you should use  

mediaTime

.

summary

Being able to access the camera in the browser without using third-party software is an incredible progress . And  

canvas

  And some of the JavaScript Combination , Cameras become fast and easy to access . It can not only use cameras , And because  

canvas

  It's super flexible , Will be able to add sexy in the future  

instagram

  Style image filter .