Iclr'22 spotlight | how to measure the amount of information in neural network weights?

This paper measures the information in the weight of neural network , Thus, the generalization ability of the network can be measured . We hope that this information measure can help improve the interpretability of deep learning , And it can be used to design algorithms to enhance network training in the future .

Address of thesis ：

https://openreview.net/forum?id=iLHOIDsPv1P

We know , Explaining the generalization ability of neural networks is a hot research in the field of artificial intelligence in recent years . The amount of information stored in the neural network is considered as an index to measure the generalization ability of the network . If we can calculate the amount of information in the neural network , Then we can explain and infer the generalization ability of the network , Unlock the black box of deep learning , And guide its optimization and Design .

In this article , We propose an algorithm that can quickly calculate the amount of data information stored in the weight of neural network , Based on this, a new information bottleneck is constructed (PAC-Bayes IB). We proved in experiments that , This kind of information measurement can explain and track the generalization ability of the network from many angles , For example, in different widths and depths , data size , Data noise level , Batch size, etc . also , Using this kind of information as a constraint can train a better neural network .