Thesis reading_ Multi task learning_ MMoE

Introduce

English title ：Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts
Chinese title ： Task relationship modeling in multi expert mixed multi task learning
Address of thesis ：https://dl.acm.org/doi/pdf/10.1145/3219819.3220007
field ： Deep learning , Multi task learning
Time of publication ：2018
author ：Jiaqi Ma, University of Michigan , Google
Source ：KDD
Quantity cited ：137
Code and data ：https://github.com/drawbridge/keras-mmoe
Reading time ：22.07.24

Journal entry

Multi task learning is generally used for the same input characteristics , Use a model to learn multiple tasks at the same time ． Predict multiple tags at once , This can save training and prediction time , It can also save space for storing models ．
The previous method is mainly the underlying shared network , The upper layer trains its own network for each task ． The problem with this is , If multiple tasks are not highly relevant , It is possible to pull shared parameters in different directions , Although in theory, multiple tasks can assist each other , Provide more information , But the effect of implementation is often not as good as that of a separate training model ．

Introduce

The effect of multi task learning generally depends on the correlation between different tasks ． What is put forward in this paper MMoE（Multi-gate Mixture-of-Experts） Yes, the previous method MoE Improvement ． It is mainly used to solve the problem of multi task correlation , Optimize multiple objectives at the same time ． For example, predict whether users buy and user satisfaction at the same time ．

In the course of research , The problems encountered are ： How to measure the relevance of different tasks ; If the model is not too large and complex due to multitasking ．

Article contribution

• Put forward MMoE structure , The upper network based on gating is constructed , The model can automatically adjust network parameters ．
• The method of generating experimental data is designed , In order to better measure the impact of task relevance on modeling
• Better results are achieved in the experimental data set , It solves the problem of large-scale data training in the real world

Method

The previous implementation method is shown in the figure -1(a) Shown , The underlying network Shared Bottom Shared parameters , The upper layer uses double tower or multi tower structure to adapt to different tasks ：

among k It's a specific task ,f(x) It's the underlying model ,hk It's the upper model ．

Then, as shown in the figure -1(b) Shown MoE Model （ It is also recorded as OMoE）, It uses multiple expert networks as the underlying , Use the input to calculate the gating value to set the proportion of each expert's contribution , Then send the calculated results to the upper network ．

among g It's door control ,n It's the number of experts , The formula combines the results of various experts ． For each entity , Only part of the network is activated ．

chart -1 Is the network structure proposed in this paper MMoE, And MoE The difference is that it calculates different gating and sets the proportion of experts for different tasks ．

Among them Wgk Is a trainable matrix , Used to select experts based on input ．

Each gating network linearly divides the input space into n Regions , Each area corresponds to an expert .MMoE Decide how areas managed by different door controls overlap . If an area is less relevant to the task , Then sharing experts will be punished , The gating network of tasks will learn to use different experts .

experiment

Synthetic data experiment

Synthetic data can better compare the effects of different task correlations , The experimental comparison using synthetic data is shown in the figure -4 Shown ：

• For all models , The training effect of tasks with high correlation is better
• In the case of different degrees of Correlation ,MMoE Better than OMoE and Shared-Bottom Model ． And when the correlation is consistent ,MMoE and OMoE The results are almost the same
• be based on MoE The effect of the two models is obviously better than Shared-Bottom, And convergence is faster , This explanation MoE Structure makes the model better trained ．

Real data experiment

Population income data

Use population income data , Two groups of experiments were carried out , The first group trains two tasks at the same time ： Whether the training income exceeds 50K Married or not ; The second group also trained in education and marriage ． Training data 199523． The training results are as follows ：

Large scale content recommendation

Train with hundreds of millions of recommendation data from Google ． The goal is to optimize ： Viscosity related goals , Such as click through rate and sticky time ; And satisfaction related goals ． Use specific evaluation criteria AUC and R-Squared scores． The effect is shown in the table -3 Shown ：

From the picture -6 You can see the contributions of different experts to different tasks ：