[deep learning paper notes] hnf-netv2 for segmentation of brain tumors using multimodal MR imaging

HNF-Netv2 for Brain Tumor Segmentation using multi-modal MR Imaging
Use multimodality MR Imaging segmentation of brain tumors HNF-Netv2
Published : Jan 2022
The paper ：https://arxiv.org/abs/2202.05268
Code ： no

Abstract ：
   In previous work , The author uses HNF-Net、 High resolution feature representation and lightweight nonlocal self attention mechanism , Use multimodality MR Image segmentation of brain tumors . In this paper , The author passes Add semantic recognition enhancement blocks between and within scales , take HNF-Net Extended to HNF-Netv2, Further use the obtained high-resolution features for global semantic recognition . The authors challenge in multimodal brain tumor segmentation (BraTS) 2021 Train and evaluate authors on data sets HNF-Netv2. Result display of test set , The author's HNF-Netv2 The average of Dice The score is 0.878514、0.872985、0.924919, Enhance tumor 、 The core of the tumor and the whole tumor Hausdorff distance (95%) Respectively 8.9184、16.2530 and 4.4895. The author's method obtains RSNA 2021 Brain tumor AI Challenge Award ( Split task ), In all 1250 Ranked eighth among the submitted results .

Question motive ：
   Glioma is the most common primary brain malignant tumor , Histologic subdomains that usually contain heterogeneity , Such as edema / attack 、 Active tumor structure 、 Cystic / Necrotic components and no enhanced gross abnormalities . Using multimodal magnetic resonance (MR) Imaging performs accurate and automated segmentation of these inherent sub regions , It is essential for the potential diagnosis and treatment of this disease . So , Multimodal brain tumor segmentation challenges have been carried out for many years (multi-modal brain tumor segmentation, BraTS), It provides a platform for evaluating the most advanced brain tumor sub region segmentation methods .
   With the wide application of deep learning in medical image analysis , Based on full convolution network (fully convolutional network, FCN) The segmentation method of is designed for this segmentation task , And demonstrated convincing performance in previous challenges .Kamnitsas Et al constructed a 3D Dual path CNN, namely DeepMedic, It adopts a dual path architecture , At the same time, multi-scale processing of the input image , Thus, local and global context information can be used .DeepMedic Also used 3D Fully connected conditional random fields to eliminate false positives .Isensee wait forsomeone Use normalized sum with instances leaky ReLU Activation of 3D U-Net, Combining combined loss function and region based training strategy , Achieved excellent segmentation performance .Myronenko Et al. Will be based on a variational self encoder (V AE) The reconstruction decoder of is added to 3D U-Net in , Regularize the shared encoder , And in 2018 year BraTS It has won the first place in segmentation performance . stay BraTS 2019 in ,Jiang et al. A two-stage cascade U-Net Model , Divide brain tumors into molecular regions from coarse to fine , The second stage model has more channels , And use two decoders to improve performance . The method in BraTS 2019 The best segmentation effect is achieved in the segmentation task . In the author's previous work , The author proposes a high-resolution and non local feature network (HNF-Net), Used in multimodal MR Segmentation of brain tumors in images . The HNF-Net Mainly based on parallel multiscale fusion (PMF) Module building , It can maintain high-resolution feature representation , Aggregate multi-scale context information . The model also The expectation maximization attention is introduced (EMA) modular , At the cost of acceptable computational complexity , Enhanced long-term dependent spatial context information . stay BraTS 2020 In the challenge , The author designed a two-stage cascade HNFNet, Thus, a hybrid high-resolution and nonlocal feature network is constructed (H2NF-Net), The network uses single and cascade models to segment different brain tumor sub regions . Proposed H2NF-Net stay 78 Participants won BraTS 2020 Challenge the second place in the segmented task .
   The characteristics of shallow stage have more detailed spatial information , The features in the deep stage have stronger semantic recognition ability . Due to the author's PMF modular , In the original HNF-Net in , High resolution features are well maintained , However, due to the limited global context learning ability of the current network , Semantic recognition of the obtained features may not be sufficient . To solve these problems , In this paper , The author further proposes a HNF-Netv2, To improve the segmentation performance of this challenging task . To be specific , author By adding semantic recognition enhancement blocks between and within scales ( Between scales SDE And within the scale SDE) To expand the original HNF-Net, Thus, the global context is used to enhance the current high-resolution features with semantic recognition . The author in BraTS 2021 The proposed HNF-Netv2, The results on the verification set and the test set show that the author's method has excellent segmentation performance , The ablation study on the training set shows that the proposed intra scale and inter scale SDE Block validity .

   chart 1 Shown . Example scan all modes , And attach the corresponding basic facts .NCR/NET、ED and ET Areas are in red 、 Green and yellow indicate .

Model method :

   chart 2 Shown . original HNF-Net Architecture of . In every study , Four multimodal brain MRI sequences are first connected to form a four channel input , Then it is processed on five scales .R Is the original resolution ,c by feature map The number of channels . All down sampling operations are performed through 2 Step convolution , All upsampling operations are performed through joints 1 × 1 × 1 Convolution and trilinear interpolation . It should be noted that , Because it is inconvenient to show in the figure 4D Characteristics of figure (C × D × H × W), So the author shows all the characteristic graphs , No depth information , The thickness of each feature map can display its channel number .

  HNF-Net have 5 A standard codec structure , Pictured 2 Shown . On the original scale r On , There are four convolution blocks , Two for encoding , The other two are used for decoding . On the other four scales , four PMF Modules are jointly used as high-resolution and multi-scale aggregate feature extractors . In the last PMF End of module , First, the output feature map of four scales is restored to 1/2r scale , Then they are spliced into mixed features . secondly , utilize EMA The module effectively captures remote dependency context information , And reduce the redundancy of the obtained mixed features . Last ,EMA The output of the module is through 1×1×1 Convolution and upsampling are restored to the original scale r and 32 passageway , Then it is added to the full resolution feature map generated by the encoder , For dense prediction of voxel labels . All sample operations are passed 2 Step convolution implementation , All loading operations are through joints 1 × 1 × 1 Convolution and trilinear interpolation .

   chart 3 Shown .HNF-Netv2 The architecture of . And graph 2 be similar ,r Is the original resolution ,c by feature map The number of channels . meanwhile , The author displays all figure maps without depth information , The thickness of each figure map shows its channel number . And HNF-Net comparison , The author is cascading PMF Inter scale and intra scale SDE block .

   Have proved , Learning strong resolution representation is very important for small target segmentation , For example, tumor and lesion segmentation in medical images . On this basis , Using multiscale convolution branching and full connection fusion settings PMF modular , The former can make full use of multi-resolution features , But maintain high-resolution feature representation , The latter can aggregate rich multi-scale context information . Besides , The author is in the author's HNF-Net There are many in series PMF modular , The number of branches increases with depth , Pictured 2(a) Shown . therefore , From the highest resolution stage , Through repeated fusion of multi-scale low resolution representation , Improved its high-resolution feature representation .
   Although the nonlocal self attention mechanism has a convincing ability to aggregate context information from all spatial locations and capture long-term dependencies , But because of its potential high computational complexity , Difficult to apply to 3D Medical image segmentation task . To solve this problem , The author is in the author's HNF-Net Introduced in EMA modular , It aims to incorporate the lightweight nonlocal attention mechanism into the author's model .EMA The main concept of the module ( Pictured 2 (b) Shown ) It is to operate nonlocal attention on the basis of a set of feature reconstruction , Instead of directly implementing on high-resolution feature map . Because the elements of the reconstructed base are much less than the original feature map , It can significantly reduce the computational cost of nonlocal attention .EMA The details of the module can also be found in .

   chart 4 Shown .(a) Within scale semantic discrimination enhancement block ,(b) Inter scale semantic discrimination enhancement block , Multi scale attention module structure .GAP, c〇,+〇,×〇 Respectively represents global average pooling , Series connection , Summation of elemental formula , Matrix point multiplication .

   In the proposed HNF-Netv2 in , author In cascade PMF Inter scale and intra scale SDE block . Pictured 3 Shown , The author in PMF Each convolution branch of the module is internally constructed within the scale SDE block . meanwhile , In order to control the computational complexity , The author is only on page 3 individual PMF Module and section 4 individual PMF Insert a scale between modules SDE block . Now the author will delve into the details of these key components .
   Between scales SDE Module adoption PMF Module full connection fusion block deployment , Structure is shown in figure 4 (a) Shown . author First, scale n( At present PMF The smallest branch in the module ) The characteristics of apply a global average pool (GAP) layer , Get the global context with high semantic information . then , The author samples the obtained features to PMF Resolution of each branch of the module . Considering the poor spatial information of upsampling features , Author use 1×1×1 The convolution layer reduces the number of channels to 1, Then connect it with high-resolution features . Under this setting , The author can reduce the damage to the original spatial information while , Add global semantic recognition to high-resolution features . differ inter-scale End module ,intra-scale The internal structure and diagram of each convoluted branch and module of the end block 4 Structure shown in (b). author We also use a gap layer to generate global context and apply two 1×1×1 Convolution layer adjusts the channel number . And based on cnn The prediction layer of the classification network is similar , The obtained global features collect information from various spatial locations , It has strong semantic information . therefore , The author can use these global features to reweight the input high-resolution features , So as to further enhance the global semantic recognition ability . In previous work [10,9] after , The author finally concatenated the multi-scale enhancement features as EMA Module input .

Experiments and results ：
Data and evaluation indicators ：
  BraTS The challenge celebrates 10 Anniversary of the , By the North American Radiological Society (RSNA)、 American Society of neuroradiology (ASNR) And the society for medical image computing and Computer Assisted Intervention (MICCAI) To co organize .BraTS21 Data set containing 2000 A multimodal brain MR Research (8000 individual mpMRI scanning ), Include 1521 A training 、219 A verification and 260 Test cases . The data setting is the same as that of previous years , Each study has 4 Zhang MR Images , Respectively T1 weighting (T1)、 After contrast T1 weighting (T1ce)、T2 weighting (T2) And liquid attenuation inversion recovery (Flair) Sequence , Pictured 1 Shown . all MR The size of the image is 240 × 240 × 155, Voxel spacing is 1 × 1 × 1mm3. For every study , Experts have enhanced the tumor (ET)、 Peritumoral edema (ED)、 Necrotic and non enhanced tumor core (NCR/NET) Mark voxels one by one . The notes of training research are public , The notes of the validation study and the test study will be retained , Used for online evaluation and final segmentation competition .

   In order to evaluate HNF-Netv2 Proposed in SDE Block validity , The author first used five fold cross validation to study the ablation of the training set . The author chooses 3D U-Net As a baseline model . Besides , The author has tested the original HNF-Net Performance of , Within the scale SDE block , And further test the SDE block ( Proposed HNF-Netv2). As shown in the table 1 Shown , adopt Dice Score and evaluate the segmentation performance

   surface 1：BraTS 2021 Ablation study of training device .Inter-SDE: Between scales SDE block ,Intra-SDE: Within the scale SDE block ,DSC: Dice similarity coefficient ,HD95: Hausdorff distance (95%),WT: The whole tumor ,TC: Tumor core ,ET: Enhance the tumor core .

   surface 2： The author's method is in BraTS 2021 Verify the split performance on the set .DSC: Dice similarity coefficient ,HD95: Hausdorff distance (95%),WT: Whole tumor ,TC: Tumor core ,ET: Enhance the tumor core . The average and median values of the segmentation results of all files are provided .

   surface 3： The author's method is in BraTS 2021 Segmentation performance on the test set .DSC: Dice similarity coefficient ,HD95: Hausdorff distance (95%),WT: Whole tumor ,TC: Tumor core ,ET: Enhance the tumor core . The data is provided by the challenge organizer , The average and median values of the segmentation results of all files are listed .

summary ：
   In this paper , author A method based on multi-modal analysis is proposed MR Imaging brain tumor segmentation HNF-Netv2, It adds inter scale and intra scale SDE Block to extend HNF-Net, Enhance the semantic recognition ability of features . The author in BraTS 2021 The author's method was evaluated on the challenge data set , Convincing results show , New key blocks and HNF-Netv2 The effectiveness of the .