[2022 freshmen learning] key points of the third week

1、Batch Normalization

Use BN What we need to pay attention to when ：

• Because the mean and variance will be counted in real time during training , However, historical statistical values should be used in the test , Not current , So set it up . stay pytorch You can create a model by model.train() and model.eval() Method control .( Similar to that Dropout)
• batch size Set it as large as possible , The larger the setting, the closer the mean and variance are to the real distribution of the whole data set .（ But also consider your hardware ）
• take BN Layer on the convolution layer （Conv） And activation layer （ for example ReLU） Between , And don't use bias in the convolution layer bias.

2、 Grouping convolution

Group Convolution It's about input feature map Grouping , Then each group is convoluted separately . If divided into G Group , Then the parameter quantity will be reduced to the original 1/G.

Speed up ： Theoretically, it can improve the speed of the network , But in fact, there may not be 3X3 High convolution efficiency , This is because pytorch Specifically for 3X3 The convolution of is optimized , Grouping convolution destroys this optimization .

Accuracy improvement ： Grouping can transform features into several subspaces （subspace）, Have a more comprehensive understanding of image information . Something like Transformer Inside Multi-Head Self-Attention, It's just Transformer Inside is the grouping of attention calculation , It was called “ long position ”, And the group convolution is right convolution grouping .

Transformer From natural language processing , In a real language environment , Every word and different words , Have different relationships . You can use different Attention To complement these different relationships . The above figure shows three attention, That is, three subspaces , You can better learn this relationship in subspace .

AlexNet There is also a classic discovery in , The first three lines in the above figure are GPU1 Learned filter , The last three lines are GPU2 Learned filter . You can find a major learning texture 、 Gradient information , Another major learning color information , It can be understood as different subspaces .

3、Res2Net

From the work of chengmingming teacher group of Nankai University , Characteristics of the group + The perfect combination of multiple scales . Two experiments in this paper discuss the grouping of features . It can be seen that , With the improvement of scale , Accuracy will improve , The speed will decrease . and , Grouping is greater than 4 When , Relative to grouping as 4 The improvement is not very obvious . therefore , Feature grouping is not the more the better , Increasing the number of groups will increase the calculation consumption , Need a certain balance .