Classification and method of feature fusion

1、 Definition of feature fusion

Feature fusion is an important method in the field of pattern recognition , Image recognition in the field of computer vision is a special pattern classification problem , There are still many challenges , The feature fusion method can comprehensively utilize a variety of image features , Realize the complementary advantages of multiple features , Get more robust and accurate recognition results .

2、 Classification of feature fusion

In the order of fusion and prediction , It is classified as early fusion and late fusion （Early fusion and Late fusion）

Early fusion （Early fusion）： Is to fuse features , Connect different features , Input into a model for training .（ First merge the features of multiple layers , Then the predictor is trained on the fused features , Only after full integration , Before testing .） This kind of method is also called skip connection, using concat、add operation . The representative of this idea is Inside-Outside Net (ION) and HyperNet.

Two classical feature fusion methods ：

（1）concat： Fusion of series features , Connect the two features directly . Two input features x and y If the dimension of is p and q, Output characteristics z The dimension of is p+q.

（2）add： Parallel strategy , Combine these two eigenvectors into a composite vector , For input features x and y,z=x+iy, among i Imaginary units .

Late fusion （Late fusion）： It refers to the fusion of predicted scores , The practice is to train multiple models , Each model will have a prediction score , We fuse the results of all models , Get the final prediction result .（ The detection performance is improved by combining the detection results of different layers , Before the final fusion , Start detection on the partially fused layer , There will be multiple layers of detection , Finally, multiple detection results are fused ）.

There are two kinds of ideas for this kind of research ：

（1）feature No integration ： Multiscale feture Forecast separately , Then we synthesize the prediction results , Such as Single Shot MultiBox Detector(SSD),Multi-scale CNN(MS-CNN).

Be careful ：SSD, Full name Single Shot MultiBox Detector, yes Wei Liu stay ECCV 2016 A target detection algorithm is proposed , So far, it is one of the main detection frameworks , comparison Faster RCNN There is a clear speed advantage , comparison YOLO And obviously mAP advantage （ But it has been CVPR 2017 Of YOLO9000 transcend ）.

MS-CNN：cNN Multiscale

（2）feature Pyramidal fusion , Forecast after fusion , Such as Feature Pyramid Network(FPN) etc.

Be careful ：FPN The full name is Feature Pyramid Network, That is, the characteristic pyramid network , This paper mainly aims at the multi-scale feature of the target in the image , Multiscale is very common in target detection , And corresponding to different problems, we should design different FPN.

3、 Methods of deep feature fusion and high-level feature fusion

（1） Early fusion , Using the classical method of feature fusion , In the existing network （ Such as VGG19） in , use concat or add Merge some of these layers .

variant ： use DCA Feature fusion method replaces concat and add operation .

（2） Late fusion , Pyramid network with similar characteristics （FPN） Thought , Predict the feature fusion .（ＦＰＮ Generally used for target detection , Improve the ability of small target detection ）.

Three variants ：

• YOLO2 Methods , Only in the pyramids top-down The last layer of the path predicts , Besides, there are U-Net [31] and SharpMask for segmentation, Recombinator networks for face detection, and Stacked Hourglass networks for keypoint estimation;
• YOLO3 Methods , Make predictions at every level of the pyramid ;
• FSSD Methods , Yes FPN Make minor modifications ;

feature No integration , Multiscale feature Forecast separately , Then we synthesize the prediction results .

（3） Replace the ordinary network with a network with high and low feature fusion capability , Such as Densenet.

（4） No high-level and low-level feature fusion , But on the basis of high-level feature prediction , Then the bottom features are used to adjust the prediction results .

4、DCA Method of feature fusion

The purpose of feature fusion , Is to extract features from an image , Merge into a feature that is more discriminative than the input feature .

be based on CCA（ Canonical association analysis ） Fusion method , Is to use the correlation between two input features , The calculated two transformed features have higher correlation than the input two feature sets .CCA Major deficiencies of , It ignores the relationship between class structures in the data set , So separate the classes in each set of features , In order to solve CCA Weakness , Introduced DCA.DCA Maximize the correlation between the two features and their corresponding features , At the same time, maximize the differences between different classes .

5、 The improved SSD Target detection algorithm （DeseNet）

The improvement lies in the use of Dense net Instead of the original SSD The structure of the VGG, To improve the accuracy of target detection , Improved SSD The network produces a good fit for small objects .

DeseNet（ Dense convolution network ）

6、FPN（ Characteristic pyramid ）

Feature pyramid is a commonly used structure for identifying targets with different scales , But the feature pyramid needs a large amount of computation and video memory , So in order to solve this problem , A new method of constructing feature pyramid is developed , It can reduce the additional consumption of computation and video memory .

7、YOLO

YOLOv2 There is a special transition layer in the network structure （Passthrough Layer）, Suppose the size of the last extracted feature graph is 1313, The function of the conversion layer is to convert the previous 2626 The characteristic diagram of and the 1313 The characteristic graph of （ Expand the amount of feature dimension data ）, And then merge , Then the fused feature map is used for detection . This is done to enhance the accuracy of the algorithm for small target detection . For better results ,YOLOv3 This idea has been strengthened and improved .

YOLO v3 use （ similar FPN） Upsampling and fusion practices , Integrated 3 A scale （13x13、26X26 and 52x52）, Do the detection independently on the fused feature map of multiple scales , Finally, the detection effect of small targets is significantly improved .（ Some algorithms use multi-scale feature fusion , However, it usually uses the fused single feature map for prediction , such as YOLOv2,FPN The difference is that the prediction is carried out in different characteristic layers .）