[optical flow] - data format analysis, flowwarp visualization

Optical flow

1. data format

Optical flow data is not easy to label , At least we haven't seen the method of optical flow annotation . General , Optical flow represents , The movement of the same pixel , Forward optical flow is t -> t+1 Of , So when visualizing optical flow , The outline of the object and t The time is consistent . Backward optical flow is generally defined as ：t - > t-1.

Open source optical flow data has many formats , However, generally, examples of data analysis are provided , Just read and parse by example . The data of optical flow is ：HxWx2,H,W Respectively represent the height and width of the image ,2 It means that x,y Number of moving pixels in the direction . meanwhile , The movement here is based on the current point .

Here to sintel Visualize a picture in the dataset .
The first picture is 0.5 x img1 + 0.5 x img2.
The second picture is optical flow visualization .

2. flow warp

How to check the correctness of data when you get an open source optical flow data set ？ How to combine optical flow and pictures for intuitive visualization ？

flow warp It is to sample pictures according to optical flow . Here we use cv2.remap To implement .

code:

step ：

• according to h, w Generate benchmark matrix , Be careful grid It's a HxWx2 Matrix , The last dimension 2, Express [x, y] Displacement in direction . The following is right grid Printing of partial values .
  
• grid + flow, obtain warp Required optical flow .
• Last use opencv Of remap sampling , obtain warp The image after .
• here warp The image is img1, still img2 Well ？ Should use the img2 Come on warp, because new_flow Is in img2 Position of pixels in . So the whole thing warp It's from img2 According to the forward optical flow warp To img1 The process of .

visualization ：
The first picture is img1
The second picture is img2 warp Got img1

Why? warp Got img1 There are ghosting ？
because ：new_flow = grid + flow,grid Is the base point matrix ,flow by [0, 0] when ,remap when , Will collect the original points .flow Not for 0 when , Will be based on new_flow sampling .