Deep learning - paper reading: action structural graph convolution network as-gcn

Past problems ：

Based on the fixed skeleton between joints
Capture only local physical dependencies between joints

improvement

Yes ST-GCN A big improvement of , All use graph convolution network for behavior recognition .
The difference is ST-GCN Just focus on 18 On the skeleton diagram of joint points The relationship between physically adjacent joint points . Based on the former, this paper not only focuses on the physically adjacent joint points , And pay more attention to The dependence between non adjacent joint points in physical space .

Solved the following problems ST-GCN The shortcomings of ：
1. Extract the features of joints directly connected by bones , But far away joints that may contain key patterns are ignored
2. for example , When walking , Hands and feet are closely related . although ST-GCN Try to layer a wide range of features GCN Aggregate , But in the long-term diffusion process, the node characteristics may weaken .

Innovation points

1. An encoder structure is introduced ,A-link The reasoning model (AIM), To capture the potential dependencies of specific actions , That is, directly from the action actional links, It is using actional links To capture the potential relationship between any nodes
2. Extend the existing skeleton diagram Express high-order dependencies , namely structural links, It is using structural links To catch some high order features
2. Two types of link The modules are combined into one Generalized skeleton graph , Further proposed behavior - Structural drawings and Networks , namely AS-GCN,, Will act - Action graph convolution and timing convolution are stacked together to generate a basic building block , thus Learning spatial and temporal features for behavior recognition
3. Introduced an additional The mechanism of predicting posture , adopt Capture detailed action information , To improve the accuracy of classification .
4. What this article puts forward AS-GCN The performance of two large data sets is better than many more advanced methods ; per contra ,AS-GCN It can also be carried out accurately Prediction of future posture ;

The overall structure

The network is stacked with multiple Actional-Structural Convolution and time convolution . As a network using skeleton for recognition ,
AS-GCN It can be applied to various environments . Here we will Behavior recognition is the main task , Make future posture prediction a secondary task ,
the prediction head By retaining detailed features, we can promote self supervised learning and improve recognition accuracy

The above figure is a video sequence using the new proposed in this paper Action-links and Structural-links A representation of extracted skeleton information . The yellow line connecting nodes in the figure indicates the dependency between nodes whose physical positions are not directly adjacent to each other . The thicker the yellow line is , The stronger the relationship between these two nodes , The red circle on the node represents the current state , The intensity of the motion of this joint , The larger the red circle, the darker the color , The stronger the current motion of the node .

In the second half of the network, the behavior is divided into two branches , The above branch function is behavior recognition , The following functions are behavior prediction , In the prediction branch, this paper innovatively introduces Action-links inference moudle (AIM).
AIM It consists of an encoder and a decoder , By comparing the Action-link( That is, the potential implicit dependency ) To infer , And use this to predict the location of future nodes , That is to predict future behavior . Put the relational data of the node at the previous time into the encoder and encode it first , Then the decoder decodes .

Actional Links (A-links)

** Ideas ：** When people make an action , Our actions are not necessarily just the cooperation of some neighboring nodes , It is likely to be the interaction of some joint nodes that are not physically connected . Like clapping your hands , The joint nodes of our two hands are not physically adjacent , But for the action of clapping hands , The relevance of two hands is very high . In order to capture such Non-local The connection of , We introduced Actional links, To automatically discover some potential related nodes through data .

Training A-links The module of is called ：A trainable A-link inference module (AIM), It mainly includes two parts :encoder and decoder.

The picture below is AIM Data flow graph of , In order to infer between two joint points A-link, Joint point features are concatenated into AIM The structure of the self encoder . The encoder generates A-link, The decoder is based on A-link And previous actions generate future posture predictions .

effect ：AIM In addition, it can not only predict future behavior , And it can effectively improve the accuracy of behavior recognition ( Guess it should be the role of reverse training )