Opencv optical flow prediction and remap remapping function usage

Optical flow

optical flow （ Optical flow ） It represents the motion speed and direction of each pixel in the two adjacent frames .

Suppose we have the following color space representation of optical flow ：

Optical flow color space representation

Suppose that the objects in the whole picture move to the left and up , Then you can get the following optical flow diagram .

Optical flow

Optical flow method is a method that infers the optical flow of an object by detecting the change of the intensity of image pixels over time .

Today is mainly about opencv Computing optical flow interface in cv2.calcOpticalFlowFarneback Use , And if the current frame and the predicted optical flow are known , How do we remap cv2.remap The method of obtaining the predicted image .

cv2.calcOpticalFlowFarneback function

cv2.calcOpticalFlowFarneback yes opencv Use in Gunnar Farneback The algorithm calculates the function of dense optical flow .

Algorithm paper ：https://www.ida.liu.se/ext/WITAS-ev/Computer_Vision_Technologies/Papers/scia03_farneback.pdf

flow = cv2.calcOpticalFlowFarneback(prev, next, flow, pyr_scale, levels, winsize, iterations, poly_n, poly_sigma, flags)

Function parameter ：

• prev： The current frame image , Single channel image , Color images usually require the use of cv2.COLOR_BGR2GRAY
• next： Next frame single channel image , Size and prev Agreement
• flow： Calculated optical flow diagram , and prev Same size ,CV_32FC2 type ;
• pyr_scale： The scale relationship between the upper and lower layers of the pyramid , This parameter is generally set to pyrScale=0.5, Indicates that the upper layer of the image pyramid is the next layer 2 Downsampling
• levels： The number of layers of the image pyramid ,levels = 1 This means that no extra layers will be created , Only the original image will be used .
• winsize： Average window size ,winsize The bigger it is , The more robust the algorithm is to image noise , And it can improve the detection effect of fast moving targets , But it can also cause motion blur .
• iterations： The number of iterations of the algorithm in each layer of the image pyramid
• poly_n： It is used to calculate the number of adjacent pixels expanded by polynomial at each pixel .poly_n The bigger it is , The smoother the approximate approximation of the image , The algorithm has better robustness , It will also bring more motion blur . Usually ,poly_n=5 or 7
• poly_sigma： The standard deviation of Gauss used to smooth the derivative , Used as a basis for polynomial expansion , Usually poly_n=5 when ,poly_sigma = 1.1;poly_n=7 when ,poly_sigma = 1.5
• flags： Optional parameter values OPTFLOW_USE_INITIAL_FLOW and OPTFLOW_FARNEBACK_GAUSSIAN

Function USES ：

i_t0 = cv2.imread("1_t0.jpg")
i_t1 = cv2.imread("1_t2.jpg")
g_i_t0 = cv2.cvtColor(i_t0, cv2.COLOR_BGR2GRAY)
g_i_t1 = cv2.cvtColor(i_t1, cv2.COLOR_BGR2GRAY)
flow = cv2.calcOpticalFlowFarneback(g_i_t0, g_i_t1, None, 0.5, 3, 15, 3, 5, 1.1, 0)

This calculates flow01 The flow of light , Note that the reverse optical flow is calculated here , Direction of optical flow ： Left positive right negative , Up and down . The above code calculates from t1 To t0 The flow of light , The whole body moves downward to the right , Optical flow diagram orange , The corresponding optical flow value is negative .

Suppose we get the optical flow flow, You can go through t0 Image and flow, To predict the t1 Images of moments . You need to use remap Remapping function .

cv2.remap function

cv2.remap yes opencv Remapping function of

cv2.remap(src, map1, map2, interpolation, borderMode, borderValue )

• src： Represents the original image
• map1： Express (x,y) A mapping point of a point or simply represents (x,y) Dot x value
• map2： If map1 Express (x,y) The mapping value of ,map2 It's empty , No means (x,y) Dot y value
• Interpolation: Interpolation method
• borderMode： The boundary model . When the value is BORDER_TRANSPARENT when , Represents the singular point in the corresponding source image in the target image ( outliers) Pixels will not be modified
• borderValue： Represents the boundary value , The default is 0

remap The function is actually to get a new image by modifying the position of pixels . We want to build a target image , You need to know the position of each pixel of the target image in the original image . because map Get is float, So it is possible to map to positions between multiple coordinates , And the size of the new image may change , So there is an interpolation method in the parameter .

remap In the image deformation , Image distortion and other applications will be used .

In this paper , We already have the image data of the previous frame through the above , Then there is the optical flow data of the image , You can get map. Through remapping, the data of the next frame can be recovered through optical flow prediction .

Use the code ：

def cv_warp(input, flow):
    h, w = flow.shape[:2]
    warp_grid_x, warp_grid_y = np.meshgrid(np.linspace(0, w-1, w), np.linspace(0, h-1, h))
    flow_inv = flow + np.stack((warp_grid_x, warp_grid_y), axis=-1)
    flow_inv = flow_inv.astype(np.float32)
    warped = cv2.remap(input, flow_inv, None, cv2.INTER_LINEAR)
    return warped

There are also many models that use deep learning to predict optical flow , There will be time to introduce later .