Some superficial understanding of word2vec

Recently, a friend asked word2vec What's the matter , So I reviewed the relevant knowledge again , Record some of your thoughts , Prevent forgetting ~

word2vec Is the means to obtain word vectors , It's in NNLM Improved on the basis of .
The training model is essentially a neural network with only one hidden layer .

It comes in two forms ① skip-gram： Predict the middle from both sides ② C-BOW： Predict both sides from the middle ;
Be careful , These two forms only represent two different training methods , Finally, the input layer is taken -> The weight of the hidden layer , As word vector .

During training , With CBOW For example , Suppose the corpus is “ It's a fine day today ”; The input to the model is " today God Of God really good " Six word one-hot vector, The output is a bunch of probabilities , We hope “ gas ” The probability of occurrence is the greatest .

When writing code , Usually called gensim library , The word vector can be trained by inputting the corpus .

Some small training trick：Negative Sampling, Huffman tree
Reference resources ：[NLP] Second vector Word2vec The essence of , summary word2vec（ Blog written by lab senior brother ）