Nce detail of softmax approximation

When contacting comparative learning , I saw it NCE loss, Afraid that this blog will disappear , So I specially came to record .

Reference link ：softmax Approximate of NCE Detailed explanation — carlos9310 Problems related to classification in deep learning will involve softmax The calculation of . When there are fewer target categories , Use standard softmax It's OK to calculate by formula , When there are many target categories , It is necessary to adopt the method of approximate estimation to simplify softmax Calculation of normalization in .https://carlos9310.github.io/2019/10/15/Approximating-the-Softmax/#softmax-regression

GitHub - MachineCF/Pytorch-NCE: The Noise Contrastive Estimation for softmax output written in Pytorch

Easy to understand and explain nce loss？ - You know

Problems related to classification in deep learning will involve softmax The calculation of . When there are fewer target categories , Use standard softmax It's OK to calculate by formula , When there are many target categories , It is necessary to adopt the method of approximate estimation to simplify softmax Calculation of normalization in .

Take the language model in natural language as an example , From theory to practice, the detailed explanation is based on sampling softmax The approximate method of NCE.

Theoretical review

Logical regression and softmax Regression is two basic classification models , They all belong to linear models . The former mainly deals with the problem of classification , The latter mainly deals with multi classification problems . in fact softmax Regression is the general form of logical regression .

Logistic Regression

Logistic regression model ( function / hypothesis ) by ：

Softmax Regression