RGB-D Salient Object Detection withCross-Modality Modulation and Selection

Abstract ：

in the light of RGB-D Salient object detection (SOD) problem , An effective method is proposed to Integrate and improve cross channel complementarity . The network mainly solves two challenging problems :1) How to effectively integrate information from RGB Complementary information of the image and its corresponding depth map ,2) How to adaptively select more significantly related features . First , We propose a cross modal eigen modulation (cmFM) modular , Take the depth feature as a priori , modeling RGB-D Data complementarity Relationship , Enhanced feature representation . secondly , We propose adaptive feature selection (AFS) Module to select significantly related features and suppress poor features .AFS The module adopts multi-modal spatial feature fusion , Consider the self modal and cross modal interdependencies of channel characteristics . Third , We use Position edge attention of saliency guidance (sg-PEA) modular , Encourage our network to pay more attention to areas related to saliency . The above modules as a whole , be called cmMS block , It is convenient to refine the salient features in a coarse to fine way . Plus a bottom-up inference , Subtle salient features make SOD Accuracy and edge preservation are possible . A lot of experiments have proved that , Our network is in 6 A popular RGB-D SOD The benchmark is superior to the most advanced saliency detector .

2. introduction

        First , We came up with one Cross modal characteristic modulation (cmFM) modular , Take the corresponding depth characteristics as a priori , enhance RGB Characteristic means . This is fused with execution input [30]、 Early integration [19] Or late fusion [18] The popular strategy of , The latter roughly connects or adds multimodal information . The proposed modulation design realizes the effective integration of multimodal information through feature transformation , It clearly simulates the indivisible cross modal relationship , It reduces the interference caused by the internal inconsistency of multimodal data .

      secondly , We designed adaptive feature selection (AFS) modular , The importance of different channel characteristics in self mode and cross mode is emphasized , At the same time, multi-modal spatial features are fused by gating . This is different from the past RGB-D SOD Different algorithms , Previous algorithms treat the channel characteristics of different modes equally and independently . Relax these assumptions , So that our method can Adaptively select more significantly related features from spatial features and channel features , And suppress poor features . It can also mitigate the negative effects of improper depth map capture . therefore , Our network has additional flexibility in handling different information . We also passed Introduce significance oriented position edge attention (sg-PEA) modular To emphasize the position and edge of significance , The module collects the attention weights from the predicted saliency map and saliency edge map .

        The unique feature of this method is that the feature modulation and attention mechanism can be changed from coarse to fine （coarse-to-fine） In a way that is tightly coupled . say concretely , The fusion First of all cmFM Module execution , To provide a rich feature representation . With our AFS Module coordination , Highlight relevant features to be emphasized , Redundant features are suppressed . Characteristics related to significance are defined by sg-PEA Further refinement of modules .cmFM、AFS and sg-PEA The elaborate design of the module Allows cross modal complementarity through modulation in a coarse to fine fashion 、 Select and refine , Provide precise salient features for our network . Plus bottom-up reasoning , Precise saliency features enable us to perform more accurately 、 More robust SOD.

contribution

    An effective method is proposed RGB-D SOD Methods . Effectively integrated cross complementation , Adaptive selection of saliency related features . This is achieved by designing a fusion from coarse to fine , The fusion includes

1) A cross modal feature modulation module , The optimal affine transformation parameters are learned by taking the corresponding depth features as a priori , Flexible multi-level modulation RGB features , enhance RGB Characteristic means .

2) Adaptive feature selection module , While fusing important multi-channel spatial features , Gradually emphasize the importance of self channel features and cross channel features , Effectively capture the relationship between different modes . In six popular RGB-D SOD In benchmarking , This method is always superior to the most advanced SOD Method .

3. Related work

3.1 Significance target detection

        The work of this paper ：

        1) use Depth feature as a priori Learn the optimal affine transformation parameters , Flexible multi-level modulation RGB features ;

         2) At the same time, the self modal and cross modal channel characteristics as well as the multi-modal spatial characteristics are considered , Effectively capture the relationship between different patterns .

3.2 Characteristic modulation

        suffer FiLM Inspired by the , In this paper, we adjust the multi-level feature representation based on the corresponding depth feature . Besides , A pixel based cross modal feature modulation is designed , Provides fine-grained and fine-grained control of features . 

3.3 Attention mechanism

          The application of attention mechanism is becoming more and more diversified , Such as space attention 、 Double attention 、 Pay attention to yourself 、 Multi level attention 、 Pay attention to the passage . by comparison , In this paper, the attention mechanism is used in the adaptive feature selection module , While fusing important multi-channel spatial features in a gated manner , The interdependency between self - modal and cross - modal channel characteristics is explored .