FCN - the originator of semantic segmentation (based on tf-Kersa reproduction code)

1. What is semantic segmentation?

Image semantic segmentation, in short, is to classify all pixels on an image, and mark all objects belonging to the same class as the same pixel.

What I'm going to introduce today is the first method of semantic segmentation using convolutional neural networks - FCN.

2. FCN (Fully Convolutional Networks for Semantic Segmentation)

A common convolutional neural network is connected to several fully connected layers after multiple convolutions, and the feature map generated by convolution and downsampling is mapped into a fixed-length feature vector.The general CNN structure is suitable for image-level classification and regression tasks, because they both expect the probability of the classification of the input image in the end.Common CNN networks such as VGG and ResNet finally output a 1000-dimensional vector representing the probability that the input image belongs to each class.

The FCN removes these fully connected layers, replaces them with conventional convolutional layers, and then upsamples to the size of the original image, and outputs the prediction result.

For developers, the biggest advantage is that the changes are small. From the classic classification network to FCN, it is only necessary to replace the fully connected layer.For example, the VGG16_FCN in the paper only needs to change the final fully connected layer (4096, 1, 1), (4096, 1, 1) (1000, 1, 1) into a regular convolutional layer (4096, 7, 7), (4096, 7, 7), (1000, 7, 7), and then upsample to the original image size, so that the number of channels represents the number of output categories, and the corresponding 0 on each channel does not belong toThe pixels of this class, 1 is the pixels belonging to this class.

Accuracy loss

Of course, this will also cause a loss of accuracy. After the FCN is compressed by a backbone of 32 times (after 5 stages), if it is directly upsampled, there will inevitably be discontinuous and incorrect problems at the edge of the object.Therefore, the authors of FCN also proposed feature fusion for the outputs of different stages, supplementing the location information with the output of the shallow network, and supplementing the semantic information with the output of the deep network, thus making up for the loss on the edge to a certain extent.

As shown in the figure above, the output of stage5 is directly output by 32 times upsample, which is the structure of FCN32, which has the lowest accuracy.In FCN16, the output of stage5 is directly summed by 2 times upsample and the output of stage, and then 16 times upsample is used as the final output. The accuracy of this structure will be higher than that of FCN32.The accuracy of FCN8 is a little higher.The results will be posted later.

How to upsample

At present, there are two ways of upsampling, one is deconvolution and the other is bilinear interpolation.According to the source code published by the original author, the author used bilinear interpolation, and they themselves said that after their tests, there was no significant difference in accuracy. Bilinear interpolation has no parameters to learn, and the speed will be faster.Hurry up.

Loss function

FCN is predicted pixel by pixel, so for each pixel, the Ground Truth is either 0 or 1, even if the segmented image is single-channel (the index is given according to the number of classifications), it can become One-The form of Hot, for example, the label of VOC is 21 channels.

So according to how the pixels are predicted, cross entropy can be used.Of course, later generations have improved and adopted dice loss, focal loss, etc., and will not be discussed here.

Prediction accuracy

The prediction accuracy of FCN is shown in the figure:

3. Summary

FCN is the first person to use deep learning for semantic segmentation. Compared with many new networks, the effect is indeed worse.However, most of the following networks follow the idea of ​​FCN, and add some tricks or add some new components on this basis.If you can understand FCN, you will be familiar with other segmentation networks later.

4. Implementation code

• Code published by the original authorshelhamer/fcn.berkeleyvision.org
• Personal code reproduction Runist/FCN-keras
• FCN Papers