Solution of depth learning for 3D anisotropic images

Blog for the first time , I hope you will forgive me for the poor format ..
More than one month of internship , In order to solve the problem of anisotropy during the arxiv A lot of information has been consulted on , Found two better solutions , Share it here . As for the specific implementation, it will be completed in recent months .

What is anisotropy

In regard to 3d In the process of data processing , Especially medicine 3d Images , For example, tomography images , It's all a series of 2d Images are superimposed . And if you can stack 2d Not enough images , Often cause z The resolution of the axis is much smaller than x Axis and y The problem with the axis ( for example 3d The size of the image slice is 512* 512* 8). And this problem often leads to our dataset Applied to common image segmentation / When classifying the network , The performance will be very poor .

frequently-used 3D Image segmentation / Classification of network

Before we discuss the problem of anisotropy , Let's take a look at the commonly used 3d Image segmentation / Classification of network . For now 3d Deep learning of images , The field of medicine is the most commonly used , So these kinds of networks are basically designed to solve medical problems .

3D Unet

3D Unet yes Unet Of 3d edition , and 2d Unet Compared with not much change .
link : 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.

V Net

Vnet and 3d Unet The release time is about the same . Although compared with unet Many changes , Its final performance has not been significantly improved , It may also be that there is not much comparative data at present . According to the passage , The advantage of these changes is that they can accelerate convergence The speed of , Take up less memoirs etc. . Its structure is compared with Unet There are the following changes ：

1.Vnet and Unet The biggest difference between them is , In each step ,Vnet Use ResNet Short circuit connection （ Grey arrow ）. This is equivalent to ResBlock introduce Unet.

The input of each stage is used in the convolutional layers and processed through the non-linearities and added to the output of the last convolutional layer of that stage in order to enable learning a residual function. As confirmed by our empirical observations, this architecture ensures convergence in a fraction of the time required by a similar network that does not learn residual functions.

2.Vnet Use convolution instead of pooling

Replacing pooling operations with convolutional ones results to networks that, depending on the specific implementation, can have a smaller memory footprint during training, due to the fact that no switches mapping the output of pooling layers back to their inputs are needed for back-propagation, and that can be better understood and analysed by applying only de-convolutions instead of un-pooling operations.

3.Vnet It is a network in the field of medical imaging , Its data set is a binary classification task , Therefore, it uses Dice Loss .

Using this formulation we do not need to assign weights to samples of different classes to establish the right balance between foreground and background voxels, and we obtain results that we experimentally observed are much better than the ones computed through the same network trained optimising a multinomial logistic loss with sample re-weighting

4. Data preprocessing method

random non-linear transformations and histogram matching

link : V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.

MedicalNet

MedicalNet Will be multiple 3D Medical data sets are combined into big data sets , Based on this data set, a complete 3D-ResNet A series of pre training models and corresponding transfer learning training codes .MedicalNet The pre training network provided can be migrated to any 3D Medical imaging AI Application , Including but not limited to segmentation 、 testing 、 Classification and other tasks . It is especially suitable for small data medical images AI scene , Can speed up network convergence , Improve network performance .

link : Med3D: Transfer Learning for 3D Medical Image Analysis.

The solution to the problem of anisotropy

The above three networks are all processing 3d Commonly used in medical data , But when they face data sets with anisotropic problems , It often doesn't work well . During the next internship , We will also 3dunet And medicalnet As a control group to compare the effects of different networks . At present, there is no popular solution to this problem , Here we find two solutions , It will be realized one by one in the next few months .

be based on Z-Net Solutions for

link : Z-Net: an Anisotropic 3D DCNN for Medical CT Volume Segmentation.

Infrastructure Anisotropic spatial separable convolutions
stay Z-Net in , In addition to the lower and upper sampling layers , all 3D The convolution layer is replaced by separable spatial convolution .

The proposed Z-Net is a framework for designing or modifying a 3D DCNN architecture for CT volume segmentation, where traditional 3D convolutional operations are decomposed into 2D convolutions along XYplane and a 1D convolution along Z axis. For example, a 3D convolutional kernel of size 3×3×3 is separated into a 2D convolutional kernel of size 3×3×1 and a 1D convolutional kernel of size 1×1×D, where D is the depth of the input feature map. The kernel size of 1D convolutional kernel is set to 1×1×D rather than 1×1×3 to fully extract the interslice context among all slices without significantly increase the computational cost. Such decomposition is illustrated in Fig. 4.

Proposed Z-Net It's a framework , Can be handled separately XY Features in the plane and Z Features in the plane . It can be seamlessly integrated into the current popular medical volume segmentation DCNN, namely 3D U-Net or V Net, Generate... Separately ZU-Net and ZV-net. Its performance is obviously better than Vnet or 3DUnet. Pictured 3D U-Net Image ：

Znet The advantages of
1. Increase the amount of training

With the proposed Z-Net, the original CT volume is cropped into sub-volumes of size of 512×512×8 (named Patch-512), and the stride between the successive crops is 1, which leads to (L−1) augmented training patches from each patient’s CT volume.

2. Can be in XY Maintain a complete field of view in the slice

Z-Net maintains a full field-of-view in the XY slices while it becomes smaller along the Z axis to feed into a single GPU. As sub-volume divisions will limit the effective field-of-view for the network to perceive the entire volume of a subject, it is of utter importance to keep the spatial integrity of the features as much as possible. Cropping may introduce discontinuities along edges and misalignment between adjacent patches, and this is harmful for the dense volume segmentation task. Therefore, Z-Net chooses to only crop along the Z axis, instead of cropping along all three dimensions like Patch-64 and Patch-128 for traditional 3D DCNNs.

3. Help alleviate the problem of class imbalance

Z-Net also helps mitigating the class-imbalance problems, which is common in Patch-128 and Patch-64, especially for segmentation of small organs. Take aortic CT data as an example, in which only voxels around the center are labelled as the foreground. Patch-64 and Patch-128 cropped successively from the original volume will result in a large portion of sub-volumes cropped from the border regions having no foreground at all. If the patches are sampled selectively, the model might produce more false positives along the border regions. Z-Net ensures the presence of both foreground and background in all sub-volumes.

Application of approximation method
link : 3D APA-Net: 3D Adversarial Pyramid Anisotropic Convolutional Network for Prostate Segmentation in MR Images.

be based on AH-Net Solutions for

link : 3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes.

This method is the research and implementation of another intern in the group , So I haven't done much research . The main principle seems to be that the first layer of convolution is now used 3d Convolution kernel for feature extraction , After that, the network uses 2d Convolution . This can greatly reduce the training time .

Application of approximation method
link : DeepEM3D:approaching human-level performance on 3D anisotropic EM image segmentation.