Doctoral Dissertation of the University of Toronto - training efficiency and robustness in deep learning

Thesis link ：https://arxiv.org/abs/2112.01423

The training efficiency of degree learning model is low ; They learn by processing millions of training data many times , And it needs powerful computing resources to process a large amount of data in parallel at the same time , Not sequential processing . The deep learning model also has unexpected failure modes ; They may be fooled , Make a wrong prediction .

 

In this paper , We study the methods to improve the training efficiency and robustness of the deep learning model . In the context of learning visual semantic embedding , We find that learning more information training data first can improve the convergence speed and the generalization performance of test data . We formalize a simple technique , It is called hard negative mining , Modification of the learning objective function , No computational overhead . Next , We seek to improve the optimization speed in the general optimization method of deep learning . We show that the redundant perceptual modification of training data sampling improves the training speed , An effective method for detecting the diversity of training signals is developed , Gradient clustering . Last , We study the antagonism robustness in deep learning , And the method of realizing the maximum antagonism robustness without using additional data training . For the linear model , We prove that the maximum robustness is guaranteed only by properly selecting the optimizer , Regularization , Or architecture .