MIT doctoral dissertation optimization theory and machine learning practice

Machine learning is a method of extracting prediction models from data , Thus, the prediction can be generalized to the technology of unobserved data . The process of selecting a good model based on known data sets needs to be optimized . To be specific , The optimization process generates a variable in the constraint set to minimize the goal . This process includes many machine learning channels including neural network training , This will be the main testing ground for our theoretical analysis in this paper . In all kinds of optimization algorithms , Gradient method has become the dominant algorithm in deep learning because of its high dimensional scalability and the natural limitations of back propagation . However , Although gradient based algorithms are popular , But our theoretical understanding of this algorithm in machine learning environment seems to be far from enough . One side , Within the existing theoretical framework , Most of the upper and lower bounds are closed , The theoretical problem seems to have been solved . On the other hand , It is difficult for theoretical analysis to produce faster algorithms than the experience found by practitioners . This paper reviews the theoretical analysis of gradient method , It points out the difference between theory and practice . then , We explained why the mismatch occurred , And through the development of theoretical analysis driven by empirical observation , Some initial solutions are proposed .

Thesis link ：https://dspace.mit.edu/bitstream/handle/1721.1/143318/Zhang-jzhzhang-PhD-EECS-2022.pdf?sequence=1&isAllowed=y