CaEGCN: Cross-Attention Fusion based Enhanced Graph Convolutional Network for Clustering 2021

problem ： Existing deep clustering methods often ignore the relationship between data .




This paper proposes a deep clustering framework based on cross attention —— Enhanced graph convolution network based on cross attention fusion (CaEGCN) , The network consists of four main modules : Cross attention fusion module , Innovatively self coding the content related to individual data (CAE) And a graphic convolutional self coding module related to the relationship between layer by layer data (GAE) Connect ; And self-monitoring model , The model highlights the identification information of clustering tasks . The cross attention fusion module integrates two heterogeneous representations ,CAE The module complements GAE Content information of the module , Avoided GCN Over smoothing problem . stay GAE Module , Two new loss functions are proposed , Reconstruct the content of data and the relationship between data respectively . Last , Self monitoring module constraints CAE and GAE The distribution represented by the middle layer is consistent .

1） A deep clustering framework based on end-to-end cross attention fusion is proposed , Among them, the cross attention fusion module creatively connects the automatic picture coding module and the automatic content coding module ;

2） A cross attention fusion model is proposed , Pay attention to the weight distribution of the fused heterogeneous representation ;

3） In the graph convolution automatic coding module , Propose to reconstruct the content and relationship of data at the same time , Effectively enhanced CaEGCN Clustering performance ;

CROSS-ATTENTION FUSION BASED ENHANCED GRAPH CONVOLUTIONAL NETWORK

The overall framework is as shown in the figure ：

It consists of four main modules ： Self encoder module for extracting content information ; be based on GCN Self encoder module , Used to exploit the relationship between data ; Cross attention module , Used to connect the above two modules , The multi-level adaptive fusion strategy supplements effective content information as much as possible in the transmission process ; And a self-monitoring module for constraining the distribution consistency of the middle layer representation .

A. Constructing the Graph

Construct raw data X The method of drawing is similar to SDCN equally ,KNN Method .

B. Content Auto-encoder Module (CAE)

Adopt the full connection layer of the foundation , Loss adopts reconstruction loss , restructure X.

C. Cross-Attention Fusion Module
Cross attention fusion mechanism is used to integrate CAE The content information learned and GAE Integrate the learned data relationships .

The cross attention fusion mechanism is defined as ：
$$R=F_{att}(Q,K,V)\text{\hspace{1em}(1)}$$
​ Q：query,K：key,V：value

The initial input of the cross attention fusion module Y Defined as ：
$$Y=\gamma Z_l+(1-r)H_l\text{\hspace{1em}(2)}$$
$Z_l,H_l$ Respectively GAE And CAE The first l Layer output .$\gamma$ It's a trade-off parameter , In this experiment, it is set to 0.5.

Attention mechanism calculation ： First calculate the fusion query And fusion key The similarity between ：
$$s_{ab}=q_a\ast k_b\text{\hspace{1em}(3)}$$
Then proceed on the above basis softmax Normalized acquisition $a_{ab}$：
$$a_{ab}=softmax(s_{ab})=\frac{exp(s_{ab})}{{\sum }_{a=0}^{D_{att}}exp(s_{ab})}\text{\hspace{1em}(4)}$$

Cross attention fusion mechanism R The final output is ：
$$r_a=\sum _{b=0}^N{a}_{ab}{v}_b\text{\hspace{1em}(5)}$$
To further perceive different aspects of data , The multi head mechanism is introduced ：
$$R^m=F_{att}(Q_m,K_m,V_m),m=1,2,3,...,M.\text{\hspace{1em}(6)}$$
among $Q_m=W_m^{q}Q$,K,V Empathy .
$$R=W\cdot Concat(R_1,...,R_M)\text{\hspace{1em}(7)}$$
among Concat（·） Represents the matrix concatenation operation . This is the so-called multi head mechanism and cross attention fusion module .

D. Graph Convolutional Auto-Encoder Module (GAE)

The output obtained from the previous cross attention fusion module is represented R As GAE The input of , Perform spectral convolution .
$$Z_l=GAE(R_{l-1},A)=a_l({\hat{D}}^{-1/2}\hat{A}{\hat{D}}^{-1/2}{R}_{l-1}{U}_L)\text{\hspace{1em}(8)}$$
Loss function reconstruction $\tilde{A}=Sigmodi(Z_L^T{Z}_L)$ and $\tilde{X}=Z_L$.

E. Self-Supervised Module

In the previous study, it is difficult to judge whether it is the optimal representation of clustering , We need to give an optimization goal of clustering .

Here and such SDCN Like the previous methods Student’s t-distribution As a real label , The second normalization is used as the target label . Yes CAE and GAE At the same time, optimize .