Embedding cutting-edge understanding

Content from https://mp.weixin.qq.com/s/j34nJGomvR23ZJiqIFMoAQ

Q: With massive sparse features, how to find a good feature Embedding expression?
(1) For Item Embedding in sequence behavior, what kind of Embedding expression is better?
(2) For the recommendation model of non-behavioral sequences, with regard to feature Embedding, the usual practice is to use the Embedding Size of the feature as a super-parameter, and manually test to find a good Embedding size.However, is there a better way?
A1: Res-embedding for Deep Learning Based Click-Through Rate Prediction Modeling.
Res-embedding first proved theoretically that the generalization error of the neural network CTR model is closely related to the distribution of Items in the Embedding space. If the Items with similar user interests, the smaller the envelope radius in the Embedding space, the smaller the envelope radius.That is to say, the more compact the items of the same interest are in the embedding space, and the smaller the cluster radius is, the smaller the model generalization error is, that is, the better the model's generalization ability.This conclusion is very meaningful.Because this conclusion can be used to constrain Item Embedding in the training process to make it meet certain conditions, so as to increase the model ability.On the basis of this conclusion, Res-embedding proposes a more general method: For Item Embedding with similar user interests, we let it consist of two parts superimposed, one is the interest center shared by all Items belonging to this interestCentral Embedding, the other is the residual Residual Embedding of the Item itself.

A2: Neural Input Search for Large Scale Recommendation Models (NIS).
First imagine a relatively perfect feature Embedding allocation scheme. If it exists, it should look like this: For high-frequency features, a longer Embedding size can be assigned to it, so that it can be encoded and expressed more fullyinformation.For low-frequency features, it is desirable to assign a shorter Embedding, because for low-frequency features, it appears less frequently in the training data. If a longer Embedding is assigned, overfitting is more likely to occur, which affects the generalization performance of the model.For those very low-frequency features, there is basically nothing to learn, but it will bring all kinds of noise, so we can not allocate or let them share a public Embedding.How big is the decision or search space of the scheme in the figure, it is obvious that each step has 5 choices and 4 decision steps, so the size of the decision space is 5 to the 4th power, which means that there are so many allocation schemes, and ENAS passes a certainThe AUC evaluation index performance of each allocation scheme under the validation set data and the size of the embedding space consumed by the scheme are used to evaluate the pros and cons of each decision-making scheme.We definitely encourage solutions with good performance of validation set indicators and less space consumption, and Reward of reinforcement learning is designed with this idea.Through this mode, a reinforcement learning scheme can be designed to find the optimal Embedding scheme.