AI briefing how to use loss surfaces a model integration

1. background

The paper ：Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNN

• DNN There are many local Optimalities on the loss surface of , And the local best points are isolated from each other in surface space .

• This paper finds that there is a connected region in the weight space between isolated local optima , In this area train loss Very close ,  The following figure in the middle and right .

we propose the Fast Geometric Ensembling (FGE) method that aims to find diverse networks with relatively small steps in the weight space, without leaving a region that corresponds to low test error.

2.  Reading

• What are the benefits of finding such a connected area ？

We can see that on the connected domain ,train loss  Close , loss  Surface becomes wider,  In this way, it can be generalized better on the test set . You can make a comparison , If it is the picture on the left of the above figure , If you choose 3 One of the local optima , its loss  The loss can only be in a small area , In this way, the test set may shift to the area outside the yellow .

Wide local minima cause similar losses during training and testing ; But for narrow local minima , The losses in training and testing will be very different . It means , Wide extremum has better generalization than narrow extremum

3. Method and implementation details

3.1  Method

It can be considered as a method of model integration . The key is how to find 【1】 The connected region mentioned in

• use cycle Periodic learning rate strategy （lr stay [a1, a2])

3.2  Implementation details

• According to the original SGD Way to train an initial weight model w

• With w Is the initial weight , perform cycle  Periodic learning rate learning

• Get the model at the end of each cycle , The prediction results are fused

We used a pretrained model with top-1 test error of 23.87 to initialize the FGE procedure. We then ran FGE for 5 epochs with a cycle length of 2 epochs and with learning rates [1e-5, 1e-3]