Word2vec yyds dry goods inventory

This article is my note sharing , The content mainly comes from teacher Wu Enda's in-depth learning course . ^[1]

---

Word2Vec There are two training modes .

• CBOW(Continuous Bag-of-Words Model)
• Skip-gram (Continuous Skip-gram Model)

This article is about skip-grams ^[2].
We talked about how to get word embedding Of E A method of matrix . The neural network is used for training . Of course, we need to know that method of training , The computational complexity is very large . So is there a simpler way .

---

Basic model

stay skip-gram In the model, we should establish a supervised learning model .

Here we need to choose a context vocabulary （Content） And a target vocabulary (Target).

Model process ：

one-hot→E→word embedding e→softmax→$\hat y$

• First of all, we should start from the vocabulary one-hot vector $o$ Starting with .
• We can get one word embedding matrix $E$.
• take word embedding Matrix and one-hot Multiply the vectors to get word embedding vector $e$.
• In this way, the result is sent to softmax in .
• Get the final result $\hat y$

among softmax The calculation formula of is ：

$$p(t \mid c)=\frac{e^{\theta_{t}^{\top} e_{c}}}{\sum_{j=1}^{10000}e^{\theta_{j}^{\top} e_{c}}}$$

• $\theta_{t}$ Yes and output t Related parameters , That is, an output word t What is the probability of matching the label .
• There is no offset .

The loss function uses a natural estimate of the negative logarithm .

$$\mathcal{L}(\hat{y}, y)=-\sum_{n=1}^{10000} y_{i} \log \hat y$$

softmax Then output a 10000 The vector of the dimension . The result is the probability of all possible target words .

Optimizing this model will be much simpler than that neural network we talked about before .

solve softmax Large amount of computation

$$p(t \mid c)=\frac{e^{\theta_{t}^{\top} e_{c}}}{\sum_{j=1}^{10000}e^{\theta_{j}^{\top} e_{c}}}$$

Look at softmax Calculation formula , We should know that its denominator calculation is very large . Our example vocabulary only 1 ten thousand . The amount of calculation is terrible . If the scale of those vocabularies is 100 Wan? .

One way is to split it evenly . Consider binary search . He won't tell you at once that the final result is 1 Which position in ten thousand words . It tells you in layers .

For example, the first floor tells you , He's in the back 5000 Among them . The second floor tells you , He's ahead 7500 Among them . Then the third layer will tell you layer by layer .

Another method is to count by frequency .

Consider the kind of Harman tree . The higher level is closer to the root node . Of course, there are also those with relatively low mechanical balance , Closer to the root node .

---

1.  AI Official website of China - The world's leading online AI education 、 Practice platform (deeplearningai.net) ︎

2.  [1301.3781] Efficient Estimation of Word Representations in Vector Space (arxiv.org) ︎