Efficient Video Instance Segmentation via Tracklet Query and Proposal

Abstract

VIS Our goal is to classify at the same time , Division , Track multiple target instances in the video . The present clip-level Of VIS Input a short video , Because the timing context information of multiple frames is used . The effect is obviously better than frame-level VIS. But at present, most clip-level Methods are neither end-to-end learnable , Nor can it be real-time .VIS transformer It solves the above two problems , But because of it frame-wise Dense attention calculation , Training time is too long ; and VisTR Cannot learn from end to end for multiple video segments , Manual data association is required , Before and after clips Examples of weeks tracklet link . In this paper, the EfficientVIS Training reasoning is very efficient , And end-to-end learning . The core idea is “tracklet query and tracklet proposal that associate and segment RoIs across space and time by an interative query-video interaction". And further proposed correspondence Study , Make adjacent clips Of tracklets Links can be learned .

Tracklet Query and Proposal

use tracklet queries $\{q_i{\}}_{i=1}^N$ and tracklet proposals $\{b_i{\}}_{i=1}^N$ To jointly represent each object instance in a video .tracklet query $q_i\in R^{T\times C}$ Is the number of channels C Of embedding vector ,tracklet proposal $b_i\in R^{T\times 4}$ It's a space-time Rectangle box .