Large scale visual relationship understanding

Thesis link ： The paper

Brief introduction of the paper

         This is an article AAAI 2019 The paper of , This paper aims at a large-scale visual understanding problem , In fact, it deals with the wide distribution of visual relations and the imbalance of data . This paper develops a new relationship detection model , Embed objects and relationships into two vector spaces , At the same time, it retains the distinguishing ability and semantic affinity . This article has learned a visual and semantic module , Mapping the features of two forms to a shared space , In this space , Matching feature pairs must distinguish those that do not match , At the same time, similar feature pairs should be as close as possible .

Paper notes

①、 Object categories are usually semantically related , This connection is more subtle for the relationship between objects ( The first time I read this sentence, I was a little confused , Later, I read the examples given in the following article ).<person,ride,horse> and <person,ride,elephant> The image features should be similar ( People ride an animal ), and <person,ride,horse> and <person,walk with,horse> Although they have the same subject and object , But the image features are completely different . Here we are talking about relationship recognition with object,subject On condition that , however object recognition Independent of relationship .

②、Visual Module The design of is mainly intended to object and subject Independent of relationship Space , At the same time, it involves object and subject Of relationship It also contains the characteristics of these two objects .

③、 Network structure

          The main design idea is to make <object,subject> Independent of relationship Study , but relationship And <object,subject> There's a big connection , So in relationship Every step of the branch is fused object and subject Information . That is, you want to learn the mapping of visual features to two independent semantic spaces ( Objects and relationships ).

④、Semantic Module

         The purpose of this module is to map word vectors into an embedded space , This embedded space is more different than the original word vector space , While maintaining semantic similarity . As object / It is important that relational tags provide a good word vector representation , Because it provides an appropriate initialization that is easy to tune . About word vector The choice of , Initial use Pretrained word2vec embeddings, And then use Relationship-level co-occurrence embeddings To deal with , Maximize  P (P |S, O) 、 P (S|P, O) and P(O|S, P), Is to maximize the basis of <object,predict,subject> Two of them determine the distribution of the other .