Researcher of Shangtang intelligent medical team interprets organ image processing under intelligent medical treatment

outline

Part 1 Organ image segmentation technology under intelligent medical treatment

● Introduction

●  Radiotherapy endangering organ segmentation framework against sample imbalance and shape constraints

●  Multi organ segmentation based on cooperative training and average model

● What's next

Part 2  Q&A

Organ image segmentation technology under intelligent medical treatment

1. Introduction

radiotherapy （Radiotherapy） Is the mainstream treatment of cancer . According to the data of the World Health Organization , There are about 70% Of cancer patients need radiotherapy . Radiotherapy uses high-energy radiation to penetrate the patient's body , To destroy the cancer cells DNA structure , To completely kill cancer cells . But high-energy radiation is also harmful to normal tissues and organs , It will inevitably damage some normal organs around the tumor . So in the stage of making radiotherapy plan , Radiologists need to accurately delineate tumor targets , And the outline of the surrounding dangerous organs , Ensure that the radiation dose is mainly concentrated in the tumor target area , At the same time, protect the endangered organs from being damaged by too much radiation .

In the workflow of radiotherapy , First of all CT Analog positioning of , The purpose is to simulate the position and range of radiation exposure during radiotherapy ： Position the patient , Then proceed CT scanning , At the same time, reference marks will be made on the patient's body surface for positioning . Then doctors and physicists will locate CT The target area of tumor and the dangerous organs are outlined .

After sketching , The physiologist will design the treatment plan . It mainly includes the design of the important parameters such as the angle of the radiation field and the measurement . For example, determine how many rays to use 、 From which angle each ray enters 、 How much dose and so on . After the plan is made , The physicist will verify the feasibility of the plan , Finally, the treatment plan is transmitted to the accelerator , Treat the patient . It can be seen that radiotherapy is a complex technology , Technical personnel from multiple positions are required to cooperate at the same time , To successfully complete the treatment .

At the present stage, the delineation of tumor targets and dangerous organs occupy a lot of time and energy of doctors and physicists . Take nasopharyngeal carcinoma for example , Nasopharyngeal carcinoma is sensitive to radiation , Therefore, radiotherapy is the most important early treatment for nasopharyngeal carcinoma . At present, when making radiotherapy plan in clinic , exceed 20 Normal organs need to be protected , This includes the brain stem 、 Eyeball 、 Spinal cord, etc , We call these normal organs dangerous organs （Organs-at-risk）, The doctor needs to be in CT Draw the outline of each dangerous organ on the . At present, manual sketching is mainly used in clinical practice , The whole process takes 2-5 Hours ; At the same time, organ delineation has certain requirements for anatomical knowledge , People who are familiar with clinical knowledge can sketch . So organ mapping takes up a lot of time for doctors , As a result, patients usually have to wait a long time before they can get treatment .

In addition, due to the influence of subjective factors , Different doctors treat the same patient and the same doctor at different times , The sketching results may not be completely consistent . If the outline of the organ is not accurate , It may lead to some serious complications . For example, some patients after radiotherapy , Dry mouth may occur 、 Eating difficulties and other symptoms , Serious may also suffer from encephalomyelitis . So an accurate and robust AI Assist in endangering the organ segmentation system , It can significantly improve the efficiency and quality of radiotherapy , Effectively solve the pain points of current radiotherapy doctors , At the same time, it can shorten the waiting time of patients , So that more patients receive higher quality radiotherapy at the same time .

2.  Radiotherapy endangering organ segmentation framework against sample imbalance and shape constraints

This work focuses on the head and neck CT An outline of a dangerous organ . The organs of the head and neck have very complex anatomical structures , During radiotherapy, there are 20 Multiple normal organs need to be protected ; The size and shape of the organs vary greatly ; because  CT Limitations of imaging , Some soft tissue organs have low contrast , Causing some organs to CT There is no obvious boundary on the .

First, we counted the contents of each organ voxel The number of , Find smaller organs , Such as lens 、 Optic nerve 、 Optic chiasma 、 Pituitary gland , There are dozens to hundreds voxel; Larger organs , For example, parotid gland , Temporal lobe , There are tens of thousands of voxel, The volume difference between large and small organs is thousands of times . In addition, on three-dimensional medical images , The largest is background Part of , So there is an extreme imbalance between the organs and between the organs and the background , Will greatly affect the training of neural networks .

Existing methods usually treat large organs and small organs equally , This will lead to poor segmentation accuracy of small organs . There are also some methods that can be used in Loss Function Give higher weight to small organs , But this method can not completely solve the problem of imbalance . In addition, due to CT Imaging characteristics , Some soft tissue organs , For example, the optic nerve crosses in CT Only half of the boundary is clear .

under these circumstances , our motivation Is to design a framework to imitate the doctor's outline . First, the doctor will outline the larger organs on a normal scale , For small organs , The doctor will first determine the location of the small organs , Then enlarge the image , Focus on the area around the small organs , Make a more accurate sketch , So as to solve the problem of scale imbalance . Our framework consists of two phases , The general idea is to locate the small organs first , Then focus on the small organs nearby context , Solve the problem of imbalance .

FocusNetV2 There are two parts , The first part is divided network , The second part is the counter self encoder . The partition network is divided into three sub networks , Main partition network S-Net 、 The localization network of small organs SOL-Net And the segmentation network of small organs SOS-Net . The main segmentation network is responsible for the segmentation of all organs , At the same time, we also learn multi-scale feature , As input for subsequent operations .

●  Divide the network

S-Net The main structure of is a 3D Of unet , But we found the original unet The performance on this task is not particularly ideal , The main reason is primitive unet adopt 4 Times of down sampling to get high-level feature , Then, the spatial resolution is gradually restored through symmetric up sampling operation , The same resolution will be connected across layers feature The fusion , To compensate for the loss of detail caused by downsampling .

But the head organs in this task are small and many , So it's going on CT Only a relatively small dose can be accepted during scanning . Therefore, the data resolution can only reach about 3mm , This causes some small organs to be located only in the adjacent 1~3 individual slice In which . If you do too much down sampling , It will lead to the loss of spatial details of small organs , This has a disastrous effect on small organs .

So we only do it twice down sample , So as to reduce the loss of details of small organs , But this will bring new problems ： The receptive field of the Internet has also been greatly reduced , This will lead to network learning global Information about . To solve this problem , We are  encoder and decoder In between DenseASPP modular , To expand the receptive field of the network , At the same time, I joined ResBlock、SE module And other components to further improve performance .

After using the above network design method , The imbalance between large and small organs has not been solved . We observed the doctor's zoom in The operation of , And object detection ROI Pooling The operation is very consistent , So we simulate the operation of doctors through the localization network and segmentation network of small organs .

First, the location of each small organ is determined by the positioning network , Then cut out each small organ individually , Do two types of segmentation in the segmentation network . Unlike natural images , The position, size and shape of the same organ of different people are relatively consistent , So using keypoint It is a reasonable way to make a position by regression .

The input to the location network is  S-Net Output of the last pass feature , Returning target Is the central position of every small organ Probability Map, Think of it as a three-dimensional Gaussian Heat Map , We use it MseLoss To optimize the positioning network , After getting the location of each small organ , Take the area around the small organ as the area of interest , At the same time Feature Map、 Located Heat Map And the original CT Images contact Get up and do SOS-Net The input of , Finally, we can get the segmentation result of each small organ . Each of these small organs has a separate, segmented network , Mutual interference .

Besides , We found that the original CT Images are added to do refine , It can also improve the performance to a certain extent . For segmented networks , We use weighted focal loss as well as diceloss As a loss function . these two items. loss It can play a certain role in solving the imbalance problem .

In order to solve CT The problem of fuzzy boundary , You need to design a good shape constraint , We think that a good shape regularization term should have two characteristics ： The first point , It should be able to represent shape information in a derivable way , So we can monitor the signal BP Back to the split network ; Second point , It can distinguish subtle differences between different shapes , That is to say, the monitoring information should be as sensitive as possible to changes in shape .

●  Counter self encoder

We use autoencoder Encode shape information , By minimizing the predicted shape , Provide proper regularization supervision . In order to better measure the similarity of shapes , We put forward the strategy training of confrontation training autoencoder .

First, we conducted experiments on the clinical data set collected internally , This data set contains 1100 Multiple cases of head and neck CT ,22 A dangerous organ , We can see that our method is compared with other state-of-the-art Methods , The accuracy has been greatly improved .

From the following visual results, we can see that our method is the best for the segmentation of visual intersection .

On public datasets , Our method has also achieved the best results so far .

Paper title ：

FocusNetv2: Imbalanced large and small organ segmentation with adversarial shape constraint for head and neck CT images

Address of thesis ：

https://www.sciencedirect.com/science/article/pii/S136184152030195X

3.  Multi organ segmentation based on cooperative training and average model

This work mainly focuses on the problem of missing organ labeling . The cost of medical image annotation is actually very high , So in most scenarios, only some of the organs are labeled .

With LiTS Data sets, for example , Only liver organs are labeled , Other organs are actually treated as background . Allied , KiTS Only the kidney is labeled in the dataset , Pancreas Only the pancreas is labeled in the dataset , We define this data set as few-organ datasets. Most of the time in clinical , We need to segment all the organs for comprehensive analysis .

How to use it few-organ datasets Learn the unified multi organ segmentation model ？

Existing methods ：

● Train and deploy a single model for each subset of organs

For each subset To learn a model , Use multiple models to do inference. The disadvantage is low computational efficiency ; The spatial relationship between different organs can not be well utilized .

● self-training： Based on the first method , Generate false tags for each missing organ , Get a fully annotated dataset mixed with real tags and pseudo Tags , On this basis, learn the model of multiple organs . The disadvantage is that the generalization ability of each sub model is limited , At the same time, there are certain differences between different data sets domain gap, These pseudo tags contain a lot of noise , Damage to model training .

● ConditionCNN： Embed the category code as condition information into CNN among , Realize the segmentation of multiple organs . The disadvantage is that influence It's less efficient , There is no way to deal with a large number of processors .

Our approach is to self-training Development of , Suppress the noise contained in the pseudo tag . We found that by using co-training The strategy of , And the moving average model , It can effectively mitigate the noisy surveillance contained in the pseudo tag .

We used a kind of co-training The strategy of , Train two networks with the same structure at the same time , But they have different initialization parameters , Let the two networks learn from each other , This decoupling method can effectively prevent the continuous accumulation of errors . In order to generate more robust soft label , We will use the output generated by the average model of one network timing as the supervision signal of the other network .

In addition, only these unmarked areas will contain noise , So we use Region mask To apply supervision only to unlabel In the same area .

Experimental part , In a three organ few-organ On dataset , Our method is compared to baselinne as well as self-training Have a certain improvement , Especially for the promotion of some small organs . At the same time, our method is more efficient than  ConditionCNN Result . In more challenging Of eight organs few-organ On dataset , Our method has also achieved good results .

Paper title ：

Multi-organ Segmentation via Co-training Weight-averaged Models from Few-organ Datasets

Address of thesis ：

https://arxiv.org/abs/2008.07149

4. What’s next

● Annotation-Efficient Learning： The cost of medical image annotation is expensive , So we should think about how to efficiently use the existing marked information .

● Incorporate prior knowledge： The field of medical image relies on prior knowledge , How to integrate more prior knowledge into the learning process of neural network , Make the model more interpretable , It is a direction worth studying .

● Domain Adaptation： How to improve the generalization ability of the model across centers .

Q&A

Q: Extra use GAN To introduce shape priors as constraints ,GAN The training itself is difficult to converge , Do you need to do pre training ？

A: Autoencoder Pre training is required . Briefly introduce the training process of the whole framework ： In the first stage, we will train first S-Net, And then put the parameters fix live , Then go to training SOL-Net, Then the parameters of these two blocks face, Go again train SOS-net. Then you'll put pretraining well autoencoder Come in and train . In the end, to this 4 Two parts to do joint optimization . Training in stages , Only then can each part be trained to a better state .

Q:  Different from small organ detection before segmentation , Why is it possible to segment large organs directly without testing them ？

A:  We find that when we only use the primary partition network , The segmentation accuracy of large organs has achieved good results . That is to use the method of dealing with small organs to deal with large organs , The accuracy will not be significantly improved .

Q:  Can you explain soft label loss？

A:Soft label loss in , Let's not look at region mask, The rest is cross entropy, The one outside F It can be regarded as a monitoring signal TWA Net 1 Output ;log Inside F It can be regarded as the output of the network to be optimized , Here it is TWA Net2 Output , These two items constitute a cross entropy, Outside tao yes region mask. We will only select the areas that are not marked , Only these positions have loss Of , With marked position loss Are set to zero .

PPT download

In the official account 【 Smart things open class Pro】 The background to reply “  Medical care 04 ” Get complete PPT.

Live playback

6 month 14 Japan , The open class of wisdom and East meets with Xinhua III 、 NVIDIA plans to launch 「AI/HPC Online seminar on accelerating medical image analysis and drug development 」 The live broadcast is over . Zhang Meng, the solution architect of NVIDIA, and dongzhaohui, the solution architect of the medical industry of the third Xinhua Group, gave lectures , The themes are 《 be based on NVIDIA Clara Medical image analysis and drug development 》、《 Xinhua 3 high performance computing promotes scientific research in the medical industry 》. Interested friends can click on the original text to watch the playback .

link