【ARXIV2203】CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

1、 Research motivation

The current semantic segmentation mainly uses RGB Images , Add multi-source information as an aid （depth, Thermal etc. ） It can effectively improve the accuracy of semantic segmentation , That is, the fusion of multimodal information can effectively improve the accuracy . The current methods mainly include two ：

• Input fusion： Here's the picture a Shown , take RGB and D Data spliced together , Use a network to extract features .
• Feature fusion： Here's the picture b Shown , Two networks will be used to extract RGB and D Characteristics of , Then, feature interactive fusion is carried out in the middle of the network .

Proposed by the author CMX, Characterized by ：comprehensive interactions are considered, including channel and spatial-wise cross-modal feature rectification from the feature map, as well as cross-attention from the sequence-to-sequence perspective.

2、 The main method

CMX The main box frame of is shown in the figure below , Use two parallel backbones from RGB and X Extract features from modal input , Intermediate input CM-FRM （cross-modal feature rectification module） Make feature correction , The modified feature continues to be transferred to the next layer . Besides , The features of the same layer are also input FFM（feature fusion module） The fusion . The following will be a detailed introduction CM-FRM and FFM.

CM-FRM： cross-modal feature rectification module The structure is shown in the following figure , The size of the two input features is CHW, Then use... Separately average pooling and max pooling Pool into 1x1xC Dimension vector , Spliced as 1x1x4C, adopt MLP and sigmoid, Correct the characteristics of the upper and lower branches respectively . Subsequently, a spatial level attention calculation is performed on the features , But here the attention calculation is done “ cross ”. Final output , The following form is adopted ：$X_{out}=X_{in}+{\lambda }_C{X}_{rec}^C+{\lambda }_S{X}_{rec}^S$ . Two superparameters are used in the fusion , The experimental median values are 0.5.

FFM：feature fusion module The structure is shown in the following figure , It can be seen that , Is based on Transformer Of . Unlike other methods , Here we treat the two modes equally . But in QKV In calculation , Used 《Efficient Attention: Attention with Linear Complexities》 Where is the method , Can reduce the attention Amount of computation . stay FFN part , Adopted Depth-wise conv replace MLP, meanwhile , The residual connection adds a 1x1 Convolution can further improve the effect .

The experimental part can refer to the author's paper , There's no more talk about .