Thesis reading [distinctive late semantic graph for video capturing]

Discriminative Latent Semantic Graph for Video Captioning

Summary

• publish ：ACM MultiMedia 2021
• Code ：D-LSG
• idea： In order to strengthen object-level interactions and frame-level information（ In fact, it is for the commonly used features after processing ：2D-CNN、3D-CNN、R-CNN）, The author is divided into three main parts ：Enhanced Object Proposal： Use Graph Integrate the characteristics of time and space into latent object in ;Visual Knowledge： Aggregate the above features in latent nodes And used to predict semantic words;Sentence Validation： Use GAN The model discriminates the reconstructed visual features .

Detailed design

• Core design ： Feature fusion / The way of aggregation （ In the picture ）
  
  ps： I feel a little attention The smell of

1. Multiple Feature Extraction

• Routine treatment , Generally used 2D-CNN extract appearance（frame-level） features $V^a$,3D-CNN extract motion features $V^m$,R-CNN extract region（object） features $R$

2. Enhanced Object Proposal

• take region feature Aggregate to motion feature and appearance feature in . Use GNN Each one region feature As a node.
  
  Force an explanation according to the formula ：$v^a$ With all the region feature All have sides connected , So it aggregates all region feature Characteristics of
  
  here $\Psi$ and $\Phi$ All are Linear function Then followed a Tanh Activate .${\hat{v}}_t^m$ The calculation of is similar to

3. Visual Knowledge

• Mainly in the Graph Introduced some new nodes （latent nodes）, Aggregate the above information to generate K Candidates object visual words and K individual motion visual words（ The calculation is similar to ）
  

4. Discriminative Language Validation

• In order to make the generated caption Have better semantic information （semantic concepts）. The author generates captions restructure $P^o$ and $P^m$, Then a discriminator is used to distinguish the reconstructed visual features ${\hat{P}}^o,{\hat{P}}^m$ And real signs $P^o,P^m$.
• The specific implementation will be generated caption Through some 1D CNN+ residual Layer get sentence feature $S$, And then let $P^o$“ polymerization ”$S$ Characteristics of
  
• Give the generated visual features ${\hat{P}}^o$ And real visual features $P^o$ Scoring , Treat it as a pair, Similar to calculating their similarity
  
  
• The output score of discriminant model （ Learn to give low marks to generative features , High scores for real characteristics ）
  
• Discriminant model Loss（ The latter is the regularization term ）
  
• The loss of generative models
  

Code