Meta learning Brief

Meta Learning sketch

Let's review , Traditional machine learning or deep learning process ：

1. Identify training and test data sets
2. Determine the model structure
3. Initialize model parameters （ Usually some commonly used random distribution ）
4. Initialize optimizer types and parameters
5. Training , Until it converges

Meta Learning The goal is to learn some steps 2,3,4 Parameters of , We call it Meta knowledge (meta- knowledge)

It might as well be formalized

Suppose the data set is $D=\{(x_1,y_1),...,(x_N,y_N)\}$ among $x_i$ It's input ,$y_i$ Is the output tag

Our goal is to get a prediction model $\hat{y}=f(x;\theta )$ , among $\theta$ Represent the parameters of the model ,$x$ For input at the same time $\hat{y}$ Is the output of the prediction

The form of optimization is ：
$${\theta }^{\ast }=\arg \underset{\theta }{\min }\mathcal{L}(D;\theta ,\omega )$$
Among them $\omega$ Meta knowledge , Include ：

• Optimizer type
• Model structure
• Initial distribution of model parameters
• …

We will compare the existing data sets $D$ Divide tasks , Cut into multiple task sets , Each task set includes a training set and a test set , In the form of ：
$$D_{source}=\{(D_{source}^{train},D_{source}^{val}{)}^{(i)}{\}}_{i=1}^M$$
The optimization objective is :
$${\omega }^{\ast }=\arg \underset{\omega }{\max }\log p(\omega |D_{source})$$
That is, in the multiple task sets we segment , Find a set of configurations （ That is, meta knowledge ）, Make it optimal for these tasks .

This step is generally called Meta training (meta-training)

find ${\omega }^{\ast }$ after , It can be applied to a target task data set $D_{target}=\{(D_{target}^{train},D_{target}^{val})\}$

Carry out traditional training on this , That is to find an optimal model parameter ${\theta }^{\ast }$
$${\theta }^{\ast }=\arg \underset{\theta }{\max }\log p(\theta |{\omega }^{\ast },D_{target}^{train})$$
This step is called Meta test (meta-testing)