Semi supervised mixpatch

Self consistent regularization ： There was too little tag data before , When the generalization ability of supervised learning is poor , People generally expand training data , For example, random translation of images , The zoom , rotate , Distortion , shear , Change the brightness , saturation , Add noise, etc . Data augmentation can produce countless modified new images , Expand the training data set . The idea of self consistent regularization is , Data augmentation of unlabeled data , The new data generated is input into the classifier , The prediction results should be self consistent . That is, the sample generated by the same data expansion , The prediction results of the model should be consistent . This rule is added to the loss function , There are the following forms ,

among x Is unmarked data ,Augment(x) Said to x Do random augmentation to generate new data , θ  Is the model parameter ,y Is the result of the model prediction . Note that data augmentation is a random operation , Two Augment(x) The output is different . This L2 Loss item , Constrained machine learning model , All new images obtained by enlarging the same image , Make self consistent predictions .MixMatch Integrated self consistent regularization . Data augmentation uses random left-right flipping and clipping of images （Crop）.

The second scheme is called Minimize entropy （Entropy Minimization）【5】. Many semi supervised learning methods are based on a consensus , That is, the classification boundary of the classifier should not pass through the high-density region of marginal distribution . The specific method is to force the classifier to make low entropy prediction for unlabeled data . The implementation method is to simply add a term to the loss function , To minimize the    Corresponding entropy .

MixMatch Use "sharpening" function , Minimize the entropy of unlabeled data . This part will be introduced later .

The third scheme is called traditional regularization （Traditional Regularization）. In order to make the generalization ability of the model better , The general practice is to do L2 Regularization ,SGD Next L2 Regularization is equivalent to Weight Decay.MixMaxtch Used Adam Optimizer , And an article found that Adam and L2 There will be problems when regularization is used at the same time , therefore MixMatch Use a separate Weight decay.

A recently invented data augmentation method is called Mixup 【6】, Randomly sample two samples from the training data , Construct mixed samples and mixed labels , As new augmented data ,

 

among lambda  It's a 0 To 1 A positive number between , Represents the mixing ratio of two samples .MixMatch take Mixup It is used in both marked data and unlabeled data .

mixmatch Specific steps ：

1. Use MixMatch Algorithm , To a Batch Tag data for   x And a Batch Unlabeled data for u  Data expansion , Get one... Respectively Batch Augmented data x' and K individual Batch Of u'.

 

among T, K,   It's a super parameter. , Later on .MixMatch The data augmentation algorithm is as follows ,

 

Algorithm description ：for Loop to a Batch The marked pictures and unlabeled pictures of are expanded . To mark pictures , Only one augmentation , The label remains unchanged , Write it down as p . For unmarked data , do K Sub random augmentation （ Superparameters in the article K=2）, Input classifier , Get the average classification probability , Application temperature Sharpen Algorithm （T It's a temperature parameter , This algorithm will be introduced later ）, Get unlabeled data “ guess ” label   . At this time, the expanded tag data    There is one Batch, Expanded unlabeled data    Yes K individual Batch. take    and    Mix it up , Randomly rearrange the data set   . Final MixMatch The output of the augmentation algorithm , Yes, it will    And    Did MixUp() One of the Batch Tag data for   , as well as    And    Did MixUp() Of K individual Batch Unmarked augmented data .

. For the expanded tag data x , And unmarked augmented data u   Calculate the loss items separately ,