How to learn to get the embedding matrix e ＃ yyds dry goods inventory ＃

This article is my note sharing , The content mainly comes from teacher Wu Enda's in-depth learning course . ^[1]

---

We talked about it earlier word embedding. But how can we learn ？

Practice has proved , Building language models is learning word embedding Good way  ^[2].

Now let's build a language model to try how to predict and fill sentences .

$$I \quad want \quad a\quad glass\quad of\quad orange\quad \underline{\quad\quad\quad\quad\quad}. \\\quad \\ 4343 \quad 9665 \quad 1 \quad 3852 \quad 6163 \quad 5257\quad\quad\quad\quad\quad\quad\quad\quad$$

Make sentence prediction . Now we have marked each word of this sentence with a serial number .

$$\begin{array}{llllll} \text { I } & o_{4343} & \longrightarrow & E & \longrightarrow & e_{4343} \\ \text { want } & o_{9665} & \longrightarrow & E &\longrightarrow & e_{9665} \\ \text { a } & o_{1} & \longrightarrow & E& \longrightarrow & e_{1} \\ \text { glass } & o_{3852} & \longrightarrow & E& \longrightarrow & e_{3852} \\ \text { of } & o_{6163} & \longrightarrow & E& \longrightarrow & e_{6163}\\ \text { orange } & o_{6257} & \longrightarrow & E & \longrightarrow & e_{6257}\\ \end{array}$$

We have one-hot vector $o$ Learn to get word embedding matrix $E$, Re pass word embedding Matrix and one-hot Vector multiplication . Get the corresponding word word embedding vector $e$.

We're going to get embedding Put the vector into the neural network language model . One more pass softmax layer . And then this softmax Layer will find the most reliable answer in your word list as output .

Now analyze the dimensions .

• $[o]_{10000\times 1}$

• $[E]_{300\times 10000}$

• $[e]_{300\times 1}$

• $[h]_{6*300 \times 1}$

  h It is processed by neural network splicing , Is to predict the seventh word of the sentence , Put the first six words of the sentence together and put them into the neural network for processing .

In the process of actual operation , Due to the different degrees of protons , Therefore, not all the contents in front are spliced together every time . Instead, select a size window to predict the sliding .

For example, the setting only considers the front 4 Word . Then it comes to the step of neural network calculation ,H It will become length 1200. instead of 1800.

In this way, the neural network is used to train it . Inspired by the algorithm , Your network will constantly adjust data . Final Orange、Apple、Durian、Strawberry、Watermelon Short words will learn similar eigenvalues .

This is learning to gain word embedding matrix $E$ One of the ways .

This shows that the window we use in this example is before selection 4 Word . There are other ways to select context , For example, a few words around the vacancy 、 A few words after the vacancy .

Adapt to a skip-gram The model idea of , Then we will make a detailed introduction .

---

1.  AI Official website of China - The world's leading online AI education 、 Practice platform (deeplearningai.net) ︎

2.  Neural Probabilistic Language Models | SpringerLink ︎