[paper reading] the latest transfer ability in deep learning: a survey in 2022

• English title ：Transferability in Deep Learning: A Survey
• Chinese title ： A review of transferability in deep learning
• Paper download link ：[email protected]

---

preface

On the whole, this review is quite detailed , Transfer learning itself is widely used in artificial intelligence , Therefore, it is easy to combine with other methods , The adaptability in the third section of the original text is very critical , It is also the key content of this note , It's very theoretical , The other two parts are relatively water , Many platitudes will not be recorded . Personally, I think it is more suitable to have a certain machine learning foundation , Then people who want to consolidate and transfer relevant knowledge to read .

---

Abstract

The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

---

---

1 Introduction Introduction

• Pre training itself is a kind of transfer learning .

• Transfer learning is divided into two stages ： Preliminary training （pre-training） And To adapt to （adaptation）. The former focuses on General portability （generic transferability）, The latter focuses on specific Portability （specific transferability）.

  

---

1.1 The term Terminology

Mathematical notation	Specific meaning
$\mathcal{X}$	input space
$\mathcal{Y}$	Output space
$f$	$f:\mathcal{X}\to \mathcal{Y}$ It is a annotation function that needs to be learned
$l$	$l:\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{+}$ Is the given loss function
$\mathcal{D}$	$\mathcal{X}$ An unknown distribution on
$\hat{\mathcal{D}}$	Independent identically distributed sampling from $\mathcal{D}$ The sample of $\{{\mathbf{x}}_1,...,{\mathbf{x}}_n\}$
$P(\cdot )$	It's defined in $\mathcal{X}$ Probability of events on
$\mathbb{E}(\cdot )$	Mathematical expectation of random variables
$\mathcal{U}$	Upstream data
$\mathcal{S}$	Source field of downstream data
$\mathcal{T}$	Target areas of downstream data
$t_{\ast }$	$\ast$ The task of the field ,$\ast$ Can take $\mathcal{T},\mathcal{S},\mathcal{U}$
$\mathcal{H}$	Hypothetical space （ It can be understood as a model set ）
$h$	An assumption in a hypothetical space （ Unless otherwise specified in the following , Assumptions and models have the same meaning ）
$\psi$	Feature generator
$\theta$	Suppose the parameters
$\mathbf{x}$	Model input
$\mathbf{y}$	Model output
$\mathbf{z}$	Hidden layer feature activation generation results
$D$	Recognizer for distinguishing different distributions

Definition $1$（ Portability ）：

Given the source domain $\mathcal{S}$ Learning tasks of $t_{\mathcal{S}}$ And target areas $\mathcal{T}$ Learning tasks of $t_{\mathcal{T}}$, Portability （transferability） From $t_{\mathcal{S}}$ To acquire transferable knowledge , Put the acquired knowledge in $t_{\mathcal{T}}$ And can make $t_{\mathcal{T}}$ Of The generalization error Reduce , among $\mathcal{S}\ne \mathcal{T}$ or $t_{\mathcal{S}}\ne t_{\mathcal{T}}$.

---

1.2 summary Overview

This paper is divided into three parts ：

1. Preliminary training （Pre-training）： About some important migration model architectures , A summary of supervised and unsupervised pre training methods . This part is relatively simple , Only summarize and record the key contents .
2. adaptive （Adaptation）： A focus on Task adaptability （task adaptation） And Domain adaptability （domain adaptation）, This part is highly theoretical , In especial Domain adaptability This part summarizes a large number of theorems and statistical results , I don't think it was written by the same person .
3. assessment （Evaluation）： This paper presents a general algorithm and evaluation of open source package for migration learning , Project address at [email protected]

---

2 Preliminary training Pre-Training

2.1 Pre training model Pre-Training Model

• Generally speaking , The quality of pre training task learning directly affects the application performance of pre training model in downstream tasks .

• Generally speaking , Pre training will be carried out on a very large number of data sets , So as RNN and CNN This model architecture with local connection assumptions is usually not used as a pre training model architecture （ Because there is enough data , There is no need to simplify the model architecture ）, At present, the mainstream basic injuries are based on Transformer Large scale pre training model of . Compare with RNN and CNN,Transformer Almost no assumptions are made about the structure of the input data , That is, it can be used to deal with a wider range of data types .

• The development of pre training model in transfer learning （ Such as Figure 3 Shown ）：

  

• Figure 4 The middle and left figure shows the process of model parameter search during direct training , The right figure shows the model parameter search process after pre training migration , It means that the essence of pre training is to narrow the search scope of model parameters （ However, it seems that it can also be understood as finding a better starting point ）：

  

---

2.2 Supervised pre training model Supervised Pre-training Model

• The purpose of supervised pre training is to train the pre training model on large-scale labeled data , Then migrate to enhance downstream tasks （ Such as Figure 5 Shown ）.

  

• Standard supervised pre training is very useful in the case of labeled data volume reorganization , But sometimes it's for Opposite samples （adversarial examples） The existence of is extremely sensitive , This may affect the robustness of migration . Therefore, this section will focus on two other supervised pre training methods .

---

2.2.1 Meta learning Meta Learning

• So-called Meta learning （meta-learning）, Generally speaking, it means Learn how to learn , To improve the efficiency of migration . Its core is to Meta knowledge （meta knowledge）$\varphi$ Integrate with the model , Meta knowledge $\varphi$ Can capture different learning tasks Essential attribute （intrinsic properties）, Also known as Meta training （meta-training）. When a new task needs to be solved , Learned Meta knowledge Rescue can help target model parameters $\theta$ Quickly adapt to new tasks , This process is called Meta test （meta-testing）.

• Such as Figure 6 Shown , The left figure is to simulate Meta test Rapid adaptation conditions in the process , The meta training data is constructed into a $n$ A collection of learning tasks , Each task corresponds to a learning task $i\in [n]$, Contains training sets for adapting to this task ${\mathcal{D}}_i^{\mathrm{t}\mathrm{r}}$ And test sets for evaluation ${\mathcal{D}}_i^{\mathrm{t}\mathrm{s}}$, The picture on the right shows Meta training The objective function of is a two-level optimization problem ：
  $${\varphi }^{\ast }={\text{argmax}}_{\varphi }\sum _{i=1}^n\log P({\theta }_i(\varphi )|{\mathcal{D}}_i^{\mathrm{t}\mathrm{s}}),\text{where }{\theta }_i(\varphi )={\text{argmax}}_{\theta }\log P(\theta |{\mathcal{D}}_i^{\mathrm{t}\mathrm{r}},\varphi )\text{\hspace{1em}(1)}$$
  Here, the inner optimization is used to update the model parameters $\theta$, Outer layer optimization is used to find better Meta knowledge For migration , Meta learning The key is how to build Meta knowledge In the form of .

  

• Memory based meta learning （memory-based meta-learning）：

  The controller will receive training data ${\mathcal{D}}_i^{\mathrm{t}\mathrm{r}}$ The knowledge mined in is written into memory , And read knowledge from memory to use basic learners $\theta$ In the test data ${\mathcal{D}}_i^{\mathrm{t}\mathrm{r}}$ Make predictions on , The parameters of the controller will be constantly updated . I don't think this is a very novel method , In essence, some pre-processing data stored in advance when you are working on a project can be regarded as Memory based meta learning .

  Such as references $[150]$ Proposed Memory enhanced neural network （memory-augmented neural networks,MANN） take Binding samples represent class information （bound sample representation-class label information） Stored in external memory , Use for retrieval as a feature to predict . reference $[121]$ Is to propose another memory mechanism , The basic learner is used to provide information about the status of the current task , Meta learners interact with external memory to generate model parameters for basic learners , Learn new tasks quickly .

  Memory based meta learning For such as Less shooting classification （few-shot classification） And the downstream task of reinforcement learning is relatively advantageous , But we need to design black box architecture to merge memory mechanism , Often we don't know what is stored , And why stored things are beneficial to model migration .

• Meta learning based on Optimization （optimization-based meta-learning）：

  This method examines the better initialization of the model as meta knowledge . Such as references $[43]$ Proposed in Model agnostic meta learning （model-agnostic meta-learning,MAML） Directly find an initialization that is most suitable for migration tuning , That is, only a small amount of gradient descent iteration and a small amount of annotation data are needed to adapt to the new task . In order to learn such an initialization , For each sample task $i\in [n]$, Model $\varphi$ First of all, its training data ${\mathcal{D}}_i^{\mathrm{t}\mathrm{r}}$ The last step is $\alpha$ Gradient descent iteration ：

  $${\theta }_i=\varphi -\alpha {\nabla }_{\varphi }L(\varphi ,{\mathcal{D}}_i^{\mathrm{t}\mathrm{r}})\text{\hspace{1em}(2)}$$

  This is imitating from $\varphi$ This point starts to fine tune the model . As meta knowledge ,$\varphi$ It should have good portability , So for all tasks $i\in [n]$, Fine tuned parameters ${\theta }_i$ In the test set ${\mathcal{D}}_i^{\mathrm{t}\mathrm{s}}$ The performance should be very good ：

  $$\underset{\varphi }{\min }\sum _{i=1}^{n}L({\theta }_i(\varphi ),{\mathcal{D}}_i^{\mathrm{t}\mathrm{s}})=\sum _{i=1}^{n}L(\varphi --\alpha {\nabla }_{\varphi }L(\varphi ,{\mathcal{D}}_i^{\mathrm{t}\mathrm{r}}),{\mathcal{D}}_i^{\mathrm{t}\mathrm{s}})\text{\hspace{1em}(3)}$$

  Notice MAML The meta knowledge dimension of is too high , So references $[167]$ Use standard pre training as initialization to improve . in addition , reference $[137,145,196]$ Also on the MAML Some improvements have been made .

• Meta learning Your performance is not stable , Sometimes it's worse than the standard pre training method .

---

2.2.2 Causal learning Casual Learning

Causal learning （casual learning） Aim at Out of distribution （out-of-distribution,OOD） In the field Extrapolation （extrapolated） The migration study . Its core is to use some Causal mechanism （causal mechanisms） To capture the distribution of the complex real world , When the distribution changes , Only a few Causal mechanism change , The rest remains the same , In this way, we can get better OOD Extension . Specific as Figure 7 Shown ：

Causal mechanism Represented by the vertices of a directed acyclic graph as variables , Each edge represents a causal relationship , In this way, given the distribution of parent nodes , Get the joint distribution of each variable Non entangled factorization （disentangled factorization） form , At this time, some small changes in the distribution will only affect Non entangled factorization Local or in a sparse way . Causal learning The key problem is to obtain variables controlled by independent causal mechanisms , Here are two common methods ：

• Modular model （modular model）： reference $[56,31]$, In short, use LSTM perhaps GRU As the representation module of causal mechanism .

• Constant learning （invariant learning）： reference $[129,4]$,

  Here is the last reference $[4]$ Methods , Given data represents $\psi :\mathcal{X}\to \mathcal{Z}$, And the training environment ${\mathcal{E}}^{\mathrm{t}\mathrm{r}}$, The conditional probability of representation and output is constant if there is a classifier $h:\mathcal{Z}\to \mathcal{Y}$ At the same time, it is optimal for all environments . It can be expressed as the following optimization problem with constraints ：
  $$\begin{array}{rlrl}& {\text{minimize}}_{\psi :\mathcal{X}\to \mathcal{Z},h:\mathcal{Z}\to \mathcal{Y}}& & \sum _{e\in {\mathcal{E}}^{\mathrm{t}\mathrm{r}}}{ϵ}^e(h\circ \psi )\\ & \text{subject to}& & h\in {\text{argmin}}_{\bar{h}:\mathcal{Z}\to \mathcal{Y}}{ϵ}^e(h\circ \psi ),\forall e\in {\mathcal{E}}^{\mathrm{t}\mathrm{r}}\end{array}\text{\hspace{1em}(4)}$$
  among ${ϵ}^e(h\circ \psi )$ It means in the environment $e$ Medium predictor $h\circ \psi$ Expected error of .

---

2.3 Unsupervised pre training model

Unsupervised pre training mainly refers to Self supervised learning （self-supervised learning）, The focus is on how to build Self supervised learning Tasks are used for pre training , Methods can be divided into Generative learning （generative learning） And Comparative learning （contrastive learning） Two categories: .

2.3.1 Generative learning Generative Learning

• Such as Figure 8 Shown , Generative learning An encoder is used in $f_{\theta }$ Will disturb the input $\tilde{\mathbf{x}}$ Map to hidden layer representation $\mathbf{z}=f_{\theta }(\tilde{\mathbf{x}})$, A decoder $g_{\theta }$ Reconstruct the representation into an estimated input $\hat{\mathbf{x}}=g_{\theta }(\mathbf{z})$, The model minimizes reconstruction errors $L_{\mathrm{g}\mathrm{e}\mathrm{n}}(\hat{\mathbf{x}},\mathbf{x})$ Training . The purpose of this is to give the model the ability to generate data distribution .

  

• Generative learning Methods can be divided into two categories ： Autoregressive model （auto-regressive） And Self coding model （auto-encoding）.

  • Autoregressive model ： A cliche , Such as Figure 9 Shown , Typical language models and their variants belong to autoregressive models .

    

    Given a sequence of text ${\mathbf{x}}_{1:T}=[x_1,..,x_T]$, The training goal of language model is to maximize the conditional probability of each word segmentation ：
    $$\underset{\theta }{\max }\sum _{t=1}^T\log P_{\theta }(x_t|x_{t-k},...,x_{t-1})\text{\hspace{1em}(5)}$$
    classical GPT Model （ reference $[134]$） It belongs to autoregressive model .

  • Self coding model ： The idea is to approximate the data distribution by generating raw data according to the coding representation , common BERT Model belongs to self coding model .

    stay Figure 9 In the last picture in Cover up the language model （Masked Language Model,MLM, This is also BERT Mechanisms used in model training ）, First use $[\mathrm{M}\mathrm{A}\mathrm{S}\mathrm{K}]$ Mark in the input statement $\mathbf{x}$ Cover up some participles randomly $m(\mathbf{x})$, Then train the model according to the remaining word segmentation ${\mathbf{x}}_{\backslash m(\mathbf{x})}$ To predict these cover up participles ：
    $$\underset{\theta }{\max }\sum _{x\in m(\mathbf{x})}\log P_{\theta }(x|{\mathbf{x}}_{\backslash m(\mathbf{x})})\text{\hspace{1em}(6)}$$
    This kind of cover up is very common .

  • Autoregressive self coding hybrid model ：

    stay Figure 9 The third figure in Permutation language model （permuted language model,PLM, reference $[195]$） First, the order of the sampled statement sequence is replaced randomly , Then autoregressive prediction is carried out on the replaced series . Other classical models are trained in a similar way （ That is, both hollowed out , Change the order again ）, such as T5 Model （ reference $[136]$）,RoBERTa（ reference $[109]$）,ERNIE（ reference $[168]$）,SpanBERT（ reference $[83]$）,BART（ reference $[98]$）,GPT-3（ reference $[18]$）, Multilingual BERT（ reference $[132]$）,XLM（ reference $[91]$）.

---

2.3.2 Comparative learning Contrastive Learning

• Such as Figure 10 Shown , stay Comparative learning in , There are two different View （views）, Inquire about ${\mathbf{x}}^q$ And key ${\mathbf{x}}^k$（ From raw data $\mathbf{x}$ Build to get ）, The encoder maps different views to the hidden layer representation , The decoder further maps the hidden layer representation to Indicator space （metric space）. The goal of model learning is to minimize the same sample $\mathbf{x}$ The distance between the query and the key .

  

• Typical contrastive learning methods ：

  • Maximizing mutual information （mutual information maximization）：

    With references $[70]$ Proposed Maximize depth information （Deep InfoMax） The model, for example , It aims to learn from the relationship between high-level global context and low-level local features to obtain a transferable representation . To be specific , A given input $\mathbf{x}$, Model learning an encoder $\psi$ To maximize $\mathbf{x}$ Between input and output Mutual information , Mutual information You can train a Discriminator （discriminator）$D$ To distinguish their joint distribution from Marginal value （marginals） Product to be estimated and constrained . By using Noise comparison estimate （noise-contrastive estimation,NCE） Method , The training objectives of the model are ：
    $$\underset{\psi }{\max }{\mathbb{E}}_{\mathbf{x}\sim \mathcal{U}}\left[D(\mathbf{x},\psi (\mathbf{x}))-{\mathbb{E}}_{{\mathbf{x}}^{\prime }\sim \tilde{\mathcal{U}}}\left(\log \sum _{{\mathbf{x}}^{\prime }}{e}^{D({\mathbf{x}}^{\prime },\psi (\mathbf{x}))}\right)\right]\text{\hspace{1em}(7)}$$
    among $\mathbf{x}$ It is from the training distribution of upstream tasks $\mathcal{U}$ Input samples obtained by sampling in ,${\mathbf{x}}^{\prime }$ Is from another distribution $\tilde{\mathcal{U}}=\mathcal{U}$ Samples obtained by sampling in ,$D$ It is used to distinguish the product of joint distribution and marginal value .

    Other related work includes references $[124,178,135]$, The last one deals with the problem of zero projection .

  • Relevant location prediction （relative position prediction）：

    Here's the main thing The next prediction （next sentence prediction,NSP） Mission , For the first time in BERT Use... In the model , After that, classic ALBERT Model （ reference $[93]$） It also uses a similar pre training strategy .

  • Examples distinguish （instance discrimination）：

    Here are the references $[191]$ Of InstDisc Model , It aims to learn the transferable representation according to the relationship between samples . To be specific , Given $n$ A sample , Training encoder $\psi$ Used to distinguish different samples , That is, minimize the query of the same sample $\mathbf{q}$ And key ${\mathbf{k}}_{+}$ Distance between （ This is also called the entire sample ）, And maximize the query between different samples $\mathbf{q}$ And key ${\mathbf{k}}_{+}$ Distance between （ This is also called negative sample ）：
    $$\underset{\psi }{\min }-\log \frac{\exp (\mathbf{q}\cdot {\mathbf{k}}_{+}/\tau )}{{\sum }_{j=0}^K\exp (\mathbf{q}\cdot {\mathbf{k}}_j/\tau )}\text{\hspace{1em}(8)}$$
    among $\tau$ Is a super parameter used to control softmax The offset degree of the value ,$K$ Is the number of negative samples , In fact, this is negative sampling .

    Such as Figure 11 Shown ,InstDisc The model uses a Memory module （memory bank） To store the most recently updated representation of each key , This increases the number of negative samples , May lead to inconsistent feature representation ：

    

    Other related studies include references $[67,171,23,59,25,206]$

---

2.4 notes Remarks

Summarize the performance of all methods in this section ：

Method	Modal ductility	Task extensibility	Data efficiency	Mark the cost
Standard pre training	optimal	in	optimal	Bad
Meta learning	optimal	Bad	Bad	Bad
Causal learning	in	Bad	Bad	Bad
Generative learning	in	optimal	optimal	optimal
Comparative learning	Bad	optimal	optimal	optimal

Field description ：

• Modal ductility （modality scalability）： Can it be used for multimodal data , Text 、 picture 、 Audio-visual .
• Task extensibility （task scalability）： Can you easily migrate the pre training model to different downstream tasks .
• Data efficiency （data efficiency）： Can we get strong mobility through large-scale pre training .
• Mark the cost （labeling cost）： Whether to rely on manual data annotation .

---

3 adaptive Adaptation

3.1 Task adaptability Task Adaptation

So-called Task adaptability （task adaptation）, Given a pre training model $h_{{\theta }^0}$ And target areas $\hat{\mathcal{T}}=\{{\mathbf{x}}_i,{\mathbf{y}}_i{\}}_{i=1}^m$（ Tagged $m$ A sample is right ）, Our aim is to assume space accordingly $\mathcal{H}$ Find a specific assumption in $h_{\theta }:\mathcal{X}\to \mathcal{Y}$, Make the risk ${ϵ}_{\mathcal{T}}(h_{\theta })$ To minimize the .

generally speaking , There are two ways to adapt the pre training model to downstream tasks ：

1. Feature conversion （feature transfer）： At this time, the network layer weight of the pre training model will be fixed , Just train a fully connected network for input feature transformation .
2. fine-tuning （finetune）： At this time, the network layer weight of the pre training model is equivalent to an initial point of model training , We will continue to train and optimize its network layer weight in the sample pair of the target field .

Feature conversion Convenient operation 、 Less cost , but fine-tuning The resulting model usually performs better .

Here is a concept called Benchmark trim （vanilla finetune）, That is, directly on the target data , according to Experience risk minimization （empirical risk minimization） Fine tune the pre training model , But this method will be affected Catastrophic oblivion （catastrophic forgetting） And Negative transfer （negative transfer） The trouble with the problem ,Section 3.1.1 And Section 3.1.2 It mainly discusses how to alleviate these two problems . In addition, because the size of the model and the amount of training data are getting larger and larger ,Section 3.1.3 And Section 3.1.4 Will explore Parameter efficacy （parameter efficiency） And Data efficiency （data efficiency） The problem of .

---

3.1.1 Catastrophic oblivion Catastrophic Forgetting

• Catastrophic oblivion The first concept of the concept of Lifelong learning （lifelong learning） It is proposed that , It means that the machine learning model will gradually lose the knowledge learned from the previous task when it is trained in a new task （ reference $[86]$）.

• In the fine-tuning link , Due to the scarcity of labeled data , It may cause the model to be trained to over fit on the target data , This phenomenon is called Token collapse （representational collapse, reference $[2]$）.

• The traditional solution is to train the model with a small learning rate and Stop early （early-stopping） Strategy , But it is easy to make the model fall into local optimization , Some relatively new research methods ：

  • reference $[197]$： It is found that the portability of different network layers in the model is different , Therefore, the training methods of treating different network layers during migration should be different ;
  • reference $[112]$： Based on the above findings , Put forward Deep adaptation to the Internet （deep adaptation network,DAN）, In this network architecture Specific task header （task-specific head） The learning rate of is ten times that of other layers ;
  • reference $[74]$： Gradually from the last layer of the pre training model thaw （unfreeze） Network layer weights up to the first layer , This can effectively retain the pre training knowledge in the first layer .
  • reference $[62]$： Put forward a new idea based on Policy network （policy networks） Reinforcement learning algorithm guidance fine tuning .

• Two ways to fine tune ：

  • Domain adaptation tuning （domain adaptive tuning）： reference $[74,63,32]$

    Refers to the pre training model in the source domain , Then optimize the training samples in the target field , Usually, the pre training task is unsupervised , Such as Figure 13 Shown , reference $[74,63]$ It is proposed that when fine-tuning, we will first perform an adaptive task related to the pre training task ${\mathcal{T}}^{\prime }$ Fine tune on , Then in the target area $\mathcal{T}$ Fine tune on , Two stage fine tuning can often be achieved Multi task learning （multi-task learning） Combine technology .

    

  • Regularization tuning （regularization tuning）： reference $[86,101,103,202,79]$
    $$\underset{\theta }{\min }\sum _{i=1}^{m}L(h_{\theta }({\mathbf{x}}_i),{\mathbf{y}}_i)+\lambda \cdot \Omega (\theta )\text{\hspace{1em}(9)}$$
    among $L$ Is the loss function ,$\Omega$ Is the general form of regular term （ Such as $\Omega (\theta )=\Vert \theta {\Vert }_2^2/2$ That is to say $L_2$ The regularization ）,$\lambda$ It's the penalty factor .

    Record references here $[86]$ Proposed Elastic weight merging （Elastic Weight Consolidation,EWC） Regular terms used in ：
    $$\Omega (\theta )=\sum _j\frac{1}{2}{F}_j\Vert {\theta }_j-{\theta }_j^0{\Vert }_2^2\text{\hspace{1em}(10)}$$
    among $F$ yes Fisher information matrix estimator （estimated Fisher information matrix）,${\theta }_j$ And ${\theta }_j^0$ They are the parameters of the network layer corresponding to the fine tuned model and the pre training model , The essence of the regular term is to hope that fine tuning will not change the network layer weight of the pre training model too much .

    in fact ,EWC The assumption is that if the weights of the network layers are similar , Then their output is also similar , But because the size of neural network is getting larger , Small changes in the weight of the network layer are very easy to produce the butterfly effect , Therefore, there are two other references in the sea area $[101,103]$ Proposed DELTA And LWF, The former regularizes the difference between the output characteristics of the network layer corresponding to the pre training model and the fine-tuning model , The latter regularizes the difference between the final output results of the model , See Figure 14：

    

    Another idea of regularization is based on the essence of regularization, which is to make the model smoother , So references $[202,79]$ Directly through, in the case of small disturbance to the model input , The model output cannot change much , To force smooth models , The regular term thus constructed is ：
    $$\Omega (\theta )=\sum _{i=1}^m\underset{\Vert {\tilde{\mathbf{x}}}_i-\mathbf{x}{\Vert }_p\le ϵ}{\max }{L}_s(h_{\theta }({\tilde{\mathbf{x}}}_i),h_{\theta }({\mathbf{x}}_i))\text{\hspace{1em}(11)}$$
    among $ϵ>0$ Is a small positive number ,${\mathbf{x}}_i$ And ${\tilde{\mathbf{x}}}_i$ It is the model input before and after disturbance ,$L_s$ Is a loss function that measures the distance between the outputs of two models , For example, symmetry for classification KL Divergence or mean square error .

    The last other regularization method is based on the parameter update strategy ：

    • reference $[89]$： Random standardization （stochastic normalization）, That is, randomly use Batch regularization layer （batch-normalization layer, reference $[77]$） Of statistic （statistics） To replace the target statistics , Thus, as an indirect regularization to reduce the dependence on the target statistics .
    • reference $[96]$： Directly replace the fine-tuning model weight with part of the pre training model weight .
    • reference $[193]$： According to a certain standard, only some parameters are selected to update in fine-tuning .

    

---

3.1.2 Negative transfer Negative Transfer

• Negative transfer （negative transfer） The concept of is defined by references $[142]$ Put forward .

• reference $[187]$ Further propose to measure the relationship between different fields Negative transfer Degree method , This paper extends this idea to pre training and fine tuning .

Definition $2$（ Negative migration gap ）：

$h_{\theta }(\mathcal{U},\mathcal{T})$ Indicates data from upstream $\mathcal{U}$ The pre trained model is adapted to the target data $\mathcal{T}$ One of the models in ,$h_{\theta }(\varnothing ,\mathcal{T})$ Direct from $\mathcal{T}$ The model trained on , be Negative migration gap （negative transfer gap） Defined as ：
$$\text{NTG}={ϵ}_{\mathcal{T}}(h_{\theta }(\mathcal{U},\mathcal{T}))-{ϵ}_{\mathcal{T}}(h_{\theta }(\varnothing ,\mathcal{T}))\text{\hspace{1em}(12)}$$
Say something happened Negative transfer , if $\text{NTG}$ Being positive .

• The authors note ：

  According to the definition ,$\text{NTG}$ It measures the performance gap between the migrated model and the directly trained model （ The difference between the values of the loss function ）. if $\text{NTG}$ Being positive , That's what happened Negative transfer , This shows that the model obtained by migration is not as good as the model obtained by training directly from the target data , Then migration itself is meaningless .

• Negative transfer The reason for this ：

  • Situations where the relevance between upstream tasks and downstream tasks is not high （ The distribution drift is too large ）： reference $[109,207]$ It is the case in word segmentation prediction and document classification ;

  • Depends on the marked target data size ： It's not that the bigger the better , reference $[187,66]$ Just explain ImageNet The pre training model is used in large-scale entity discovery dataset （ such as COCO Data sets ） Your performance is not very ideal ;

  • Depends on the task adaptation algorithm ： An ideal adaptive algorithm should be able to improve the positive migration between related tasks and avoid the negative migration between unrelated tasks , But the two are actually contradictory , Specific as Figure 15 Shown ：

    

• avoid Negative transfer Methods ：（ In this part, I feel that most of the nonsense , Pure rounding words ）

  • Enhance secure migration （enhancing safe transfer）： reference $[27,78,160,186]$, It refers to identifying the harmful knowledge in the pre training model .
  • Select the correct pre training model ： reference $[199,123,198,68,23]$

---

3.1.3 Parameter efficacy Parameter Efficiency

• Parameter efficacy （parameter efficiency） What is investigated is that the pre training model will generate a complete set of model parameters for each downstream task , This is very disadvantageous in storage . One solution is to use references $[20]$ Proposed Multi task learning technology , That is, fine tune a model to solve multiple target tasks , It may be beneficial to every target task . The problem is that multiple target tasks may not be highly correlated , At this time, you still need to fine tune them separately , And Multi task learning You need to access every target task at the same time , This is in Online scene （online scenarios） Is impossible （ Goals and tasks come one after another ）.

• promote Parameter efficacy Methods ：

  • Residual tuning （residual tuning）： reference $[64,203,73,139,183]$

    The idea of residual optimization stems from the fact that fitting residual is easier than fitting function directly . Only references are recorded here $[64]$ Methods （ Other articles are too general , There is no reference significance ）, Fixed pre training model $h_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}$ The weight , Train a new model extra $h_{\mathrm{s}\mathrm{i}\mathrm{d}\mathrm{e}}$ To fit the residuals , The final model is $h(x)=\alpha h_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}+(1-\alpha )h_{\mathrm{s}\mathrm{i}\mathrm{d}\mathrm{e}}$, Be careful $\alpha$ It can change during training .

  • Parameter difference tuning （parameter difference tuning）： reference $[62,103,119]$
    $${\theta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}={\theta }_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}\oplus {\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}\text{\hspace{1em}(13)}$$
    among $\oplus$ Is the addition of element levels （ In fact, it is no different from ordinary addition ）,${\theta }_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}$ Is a fixed pre training model weight ,${\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}$ It is the residual weight of different specific tasks , Here's what we're trying to do reduction （reduce）${\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}$ To achieve parameter efficacy .

    • reference $[62]$： Use $L_0$ The penalty for （ reference $[117]$）, So that ${\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}$ sparse .
    • reference $[103]$： Use references $[2]$ Proposed FastFood Transformation matrix $M$ So that ${\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}$ Low dimension （${\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}={\delta }_{\mathrm{l}\mathrm{o}\mathrm{w}}M$）.
    • reference $[119]$： Replace addition with multiplication , namely ${\theta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}={\theta }_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}\odot {\delta }_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}$

  The difference between the two methods is that the former believes that portability stems from the output characteristics of the model , The latter believes that the portability comes from the model weight .

---

3.1.4 Data efficiency Data Efficiency

• Data efficiency （data efficiency） It is discussed that adaptation training requires a large number of labeled sample pairs , In order to reduce the dependence on training data , The resulting concept is Learn less （few-shot learning） And Zero shot learning （zero-shot learning）. There are two main ideas , One is Improve the wide applicability of the pre training model （ More knowledge and data can be integrated into the pre training model ）, Second, according to the source domain data Simply and quickly generate target domain data and its annotation .

• promote Methods of data efficacy ：,

  • Index learning （metric learning）： reference $[180,162,24]$

    It mainly refers to that the large model is easy to be fitted with a small amount of data training , But consider using some nonparametric methods , For example, the nearest neighbor algorithm can effectively deal with over fitting 、 Less shooting problem 、 Zero shot problem, etc .

    • reference $[180]$： The application of attention mechanism in obtaining weighted nearest neighbors .（ It feels too far fetched ）
    • reference $[162]$： For the classification problem , It is considered that the mean value of all sample characteristics in each category can be used as the type （prototype）, Then by looking for the nearest type To sort it out .（ In fact, it is the standard clustering algorithm ）
    • reference $[24]$： Replace the linear classifier with the classifier based on cosine distance , For less shooting learning .

  • Prompt learning （prompt learning）：

    • The authors note ：

      At first, it felt a little like software engineering agile development , And translate into Agile learning , But it seems to follow Tips More relevant .

    With references $[18]$ Super large pre training model in GPT-3 For example , In the process of fine-tuning , The model accepts input $\mathbf{x}$ And predict the output $\mathbf{y}$ The probability of is $P(\mathbf{y}|\mathbf{x})$, And in the prompt process , according to Prompt template （prompt template） Enter the original $\mathbf{x}$ Dig some Slot position （unfilled slots） obtain $\tilde{\mathbf{x}}$, The pre training model will $\tilde{\mathbf{x}}$ Make up the slot dug in $\hat{\mathbf{x}}$, And based on $\hat{\mathbf{x}}$ Output $\mathbf{y}$, The specific operation is shown in the table below ：

    name	Mark	Example
    Input	$\mathbf{x}$	I like this movie
    Output	$\mathbf{y}$	Emotional polarity ： just
    Prompt template	$f_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{t}}(\mathbf{x})$	$[X]$ In short, this is a $[Z]$ The movie
    prompt （ Unfilled ）	$\tilde{\mathbf{x}}$	I like this movie , In short, this is a $[Z]$ The movie
    prompt （ Filled ）	$\hat{\mathbf{x}}$	I like this movie , In a word, this is a good movie

    The advantage of introducing prompt is that it can deal with adaptive learning of less shooting task or zero shooting task , Especially useful in question and answer system .

    Finally, I will Prompt learning Combined with fine tuning ： reference $[151,100,189]$, Specific as Figure 17 Shown ：

    

---

3.1.5 notes Remarks

Summarize the performance of all methods in this section ：

Method	Adaptability	Data efficiency	Parameter efficacy	Modal ductility	Task extensibility
Feature conversion	Bad	in	optimal	optimal	optimal
Ordinary fine tuning	optimal	Bad	Bad	optimal	optimal
Domain adaptability tuning	optimal	in	Bad	in	optimal
Regularization tuning	optimal	in	Bad	optimal	Bad
Residual tuning	in	in	in	in	in
Parameter difference tuning	in	in	in	optimal	optimal
Index learning	Bad	optimal	optimal	optimal	Bad
Prompt learning	in	optimal	optimal	Bad	Bad

Field description ：

• Adaptability （adaptation performance）： Model performance when there is a large amount of annotation data in downstream tasks .
• Data efficiency （data efficiency）： Model performance when there is only a small amount of data in downstream tasks .
• Parameter efficacy （parameter efficiency）： When the number of downstream tasks continues to increase , Can you control the total number of parameters .
• Modal ductility （modality scalability）： Can it be used for multimodal data , Text 、 picture 、 Audio-visual .
• Task extensibility （task scalability）： Can you easily migrate the pre training model to different downstream tasks .

---

3.2 Domain adaptability Domain Adaptation

So-called Domain adaptability （Domain Adaptation）, It means that the training data in the target domain is unmarked , The training data in the source domain is labeled . So try to pre train the model in the source domain , Then try to migrate to the target field for fine-tuning . Although there is some correlation between the data of the source domain and the target domain , But there must be some differences in distribution , Therefore, the performance of the migration fine-tuning model is often poor . This phenomenon is called Distribution drift （distribution shift, reference $[133]$）, Domain adaptability It is used to eliminate the gap between the training field and the testing field Distribution drift problem .

Traditional domain adaptation methods such as Reweighting （re-weighting）、 Sample from the source domain （ reference $[165]$）、 Modeling the transformation from source domain distribution feature space to target domain distribution feature space （ reference $[53]$]）, These methods are relatively trivial , Such as references $[76,126,111]$ The study is Nuclear regeneration Hilbert space （kernel-reproducing Hilbert space） Distribution mapping method , reference $[53]$ The research is to Principal component axis （principal axes） Related to the distribution of various fields . This review focuses on Deep domain adaptability （deep domain adaptation）, That is to use the deep learning model architecture to model the adaptive module , Used to match the data distribution of different fields

stay Unsupervised domain adaptability （unsupervised domain adaptation,UDA） in , Source domain $\hat{\mathcal{S}}=\{({\mathbf{x}}_i^s,{\mathbf{y}}_i^s){\}}_{i=1}^n$ Contained in the $n$ Marked samples , Target areas $\hat{\mathcal{T}}=\{{\mathbf{x}}_i^t{\}}_{i=1}^m$ Contained in the $m$ Unmarked samples , The goal is to learn algorithms to find a hypothesis （hypothesis, It's actually mapping ）$h\in \mathcal{H}:\mathcal{X}\to \mathcal{Y}$, Minimize the target risk ：
$$\text{minimize}{ϵ}_{\mathcal{T}}(h)={\mathbb{E}}_{({\mathbf{x}}^t,{\mathbf{y}}^t)\sim \mathcal{T}}[l(h({\mathbf{x}}^t),{\mathbf{y}}^t)]$$
among $l:\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{+}$ Is the loss function . Currently on UDA The core of theoretical research is how to pass the source risk ${ϵ}_{\mathcal{S}}$ as well as Distribution distance （distribution distance） To control the target risk ${ϵ}_{\mathcal{T}}(h)$ The magnitude of , Here we mainly introduce two classical research theories $\mathcal{H}\Delta \mathcal{H}$ The divergence （Divergence, reference $[9,10,120]$） And The gap is contradictory （Disparity Discrepancy, reference $[204]$）, And how to design different algorithms based on these theories .

• The authors note ：

  there hypothesis $h$ It can be understood as the black box model of machine learning , Suppose only the true and the false , therefore $h$ The output of should be only zero one , That is, it is aimed at the problem of two classifications . risk （risk） It can be understood as the value of the loss function （ Mathematical expectation ）, On the whole, we are reducing the loss value of model training .

First use trigonometric inequality , The unequal relationship between target risk and source risk can be constructed ：

Theorem $3$（Bound with Disparity）:

Suppose the loss function $l$ yes symmetrical （symmetric） And obey trigonometric inequality , Define any two distributions $\mathcal{D}$ Assumptions on $h$ And $h^{\prime }$ Between disparity （disparity）：
$${ϵ}_{\mathcal{D}}(h,h^{\prime })={\mathbb{E}}_{\mathbf{x},\mathbf{y}\sim \mathcal{D}}[l(h(\mathbf{x}),h^{\prime }(\mathbf{x}))]\text{\hspace{1em}(14)}$$
Then the target risk ${ϵ}_{\mathcal{T}}(h)$ Satisfy ：
$${ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\mathcal{S}}(h)+[{ϵ}_{\mathcal{S}}(h^{\ast })+{ϵ}_{\mathcal{T}}(h^{\ast })]+|{ϵ}_{\mathcal{S}}(h,h^{\ast })-{ϵ}_{\mathcal{T}}(h,h^{\ast })|\text{\hspace{1em}(15)}$$
among $h^{\ast }={\text{argmax}}_{h\in \mathcal{H}}[{ϵ}_{\mathcal{S}}(h)+{ϵ}_{\mathcal{T}}(h)]$ yes Ideal joint hypothesis （ideal joint hypothesis）,${ϵ}_{\text{ideal}}={ϵ}_{\mathcal{S}}(h^{\ast })+{ϵ}_{\mathcal{T}}(h^{\ast })$ yes Ideal joint error （ideal joint error）,$|{ϵ}_{\mathcal{S}}(h,h^{\ast })-{ϵ}_{\mathcal{T}}(h,h^{\ast })|$ It's distribution $\mathcal{S}$ And $\mathcal{T}$ Between Gap difference （disparity difference）.

• The authors note ：

  The symmetry of the loss function satisfies the commutative law , namely $l(y_1,y_2)=l(y_2,y_1)$; The loss function can be seen as the difference between two vectors , So the formula $(14)$ It measures two assumptions （ The model ） The difference degree of prediction results .

In the research of domain adaptability , It is usually assumed that Ideal joint error （ That is, the sum of the loss function of the source domain task and the target domain task ） Is sufficiently small , Otherwise, domain adaptation itself is not feasible （ That is, it cannot be trained until the loss function reaches a low level , Corresponding references $[10]$ Proposed in The impossibility theorem ,impossibility theorem）, Present tense $(15)$ Only the last item needs to be examined Gap difference The numerical .

However, the target dataset label is not available , So the ideal hypothesis $h^{\ast }$ It is unknown. , therefore Gap difference It cannot be estimated directly ,$\mathcal{H}\Delta \mathcal{H}$ The divergence It is used to measure Gap difference The upper bound of ：

Definition $4$（$\mathcal{H}\Delta \mathcal{H}$ The divergence ）：

Definition $\mathcal{H}\Delta \mathcal{H}\stackrel{\Delta }{=}\{h|h=h_1\otimes h_2,h_1,h_2\in \mathcal{H}\}$ For hypothetical space $\mathcal{H}$ Of Symmetric difference hypothesis space （symmetric difference hypothesis space）, among $\otimes$ Express Exclusive or operator （XOR）, Then the distribution $\mathcal{S}$ And $\mathcal{T}$ Between $\mathcal{H}\Delta \mathcal{H}$ The divergence It can be expressed as ：
$$d_{\mathcal{H}\Delta \mathcal{H}}(\mathcal{S},\mathcal{T})\stackrel{\Delta }{=}\underset{h,h^{\prime }\in \mathcal{H}}{\sup }|{ϵ}_{\mathcal{S}}(h,h^{\prime })-{ϵ}_{\mathcal{T}}(h,h^{\prime })|$$
Specially , For the zero one loss function of binary classification problem , namely $l(y,y^{\prime })=\text{1}(y\ne y^{\prime })$, Yes ：
$$d_{\mathcal{H}\Delta \mathcal{H}}(\mathcal{S},\mathcal{T})\stackrel{\Delta }{=}\underset{\delta \in \mathcal{H}\Delta \mathcal{H}}{\sup }|{\mathbb{E}}_{\mathcal{S}}[\delta (\mathbf{x})\ne 0]-{\mathbb{E}}_{\mathcal{T}}[\delta (\mathbf{x})\ne 0]|$$

• The authors note ：

  $\mathcal{H}\Delta \mathcal{H}$ The test is the case where the two hypotheses are different （ Exclusive or operation ）. So in the second equation $\delta (\mathbf{x})$ The value of is only zero one ,$\delta (x)\ne 0$ Indicates that two assumptions are different （ That is, the prediction results of the model are different ）, The whole is the absolute value of the difference between two assumptions （ Distance ）.

  Then look at the first formula again , According to the type $(14)$ You know ,${ϵ}_{\mathcal{D}}(h,h^{\prime })$ It measures two assumptions （ The model ）$h$ And $h^{\prime }$ In the distribution $\mathcal{D}$ The difference value of the above prediction results , Absolute value measures distance , Therefore, the gap is the gap , abbreviation Gap difference .

You can sample a limited number of unlabeled samples from the source domain and the target domain $\mathcal{H}\Delta \mathcal{H}$ Divergence is estimated （ That is, use multiple groups of different models to predict the results in the source field and the target field respectively and calculate Gap difference ）, But the specific calculation optimization is very difficult . The usual practice is to train a Domain identifier （domain discriminator）$D$ To divide the samples of source domain and target domain （ reference $[9,45]$）. We assume that Recognizer family （family of the discriminators） Rich enough to contain $\mathcal{H}\Delta \mathcal{H}$, namely $\mathcal{H}\Delta \mathcal{H}\subset {\mathcal{H}}_D$（ For example, neural networks can be used to approximate almost all functions ）, be $\mathcal{H}\Delta \mathcal{H}$ Divergence can be further controlled within the range of the following formula ：
$$\underset{D\in {\mathcal{H}}_D}{\sup }|{\mathbb{E}}_{\mathcal{S}}[D(\mathbf{x})=1]+{\mathbb{E}}_{\mathcal{T}}[D(\mathbf{x})=0]|$$

• The authors note ：

  $D(\mathbf{x})=0$ Presentation sample $\mathbf{x}$ It belongs to the source field ,$D(\mathbf{x})=1$ Presentation sample $\mathbf{x}$ Belong to the target field , Therefore, two terms in the absolute value represent the probability that the prediction result is wrong , But according to Gap difference The definition of , Whether it should be a minus sign instead of a plus sign ？

This idea leads to Section 3.2.2 in Field opposition （domain adversarial） Method . Besides , If the nonparametric method is used for $\mathcal{H}\Delta \mathcal{H}$ Divergence is estimated , For example, will $\mathcal{H}\Delta \mathcal{H}$ Use a function space $\mathcal{F}$ replace , That is, derived Section 3.2.1 Medium Statistical matching （statistics matching） Method .

• The authors note ：

  According to the definition ,$\mathcal{H}\Delta \mathcal{H}$ Itself is also a hypothetical space , It can be understood as a model set or mapping group , Therefore, it can be approximated by a family of functions .

The following theorem is about Domain adaptability One of the earliest studies , It is based on $\mathcal{H}\Delta \mathcal{H}$ The general upper bound of the binary classification problem of divergence ：

Theorem $5$（ reference $[10]$）：

$\mathcal{H}$ It's a Binary hypothesis space （binary hypothesis space）, if $\hat{\mathcal{S}}$ And $$\hat{\mathcal{T}}$$ The capacity is $m$ The sample of , Then for any $\delta \in (0,1)$, There are at least $1-\delta$ The following formula holds ：
$${ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\mathcal{S}}(h)+d_{\mathcal{H}\Delta \mathcal{H}}(\hat{\mathcal{S}},\hat{\mathcal{T}})+{ϵ}_{\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}}+4\sqrt{\frac{2d\log (2m)+\log (2/\delta )}{m}}(\forall h\in \mathcal{H})\text{\hspace{1em}(16)}$$

** Theorem $5$** The drawback of is that it can only be used for binary classification problems , So references $[45]$ It is extended to the case of multi classification ：

Theorem $6$（ reference $[45]$）：

Suppose the loss function $l$ Symmetric and subject to trigonometric inequality , Definition
$$\begin{gathered} h_{\mathcal{S}}^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{ϵ}_{\mathcal{S}}(h) \\ {h}_{\mathcal{T}}^{\ast }={\text{argmin}}_{h\in \mathcal{H}}{ϵ}_{\mathcal{T}}(h) \end{gathered}$$
Respectively represent the source domain and the target domain Ideal hypothesis , Then there are ：
$${ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\mathcal{S}}(h,h_{\mathcal{S}}^{\ast })+d_{\mathcal{H}\Delta \mathcal{H}}(\mathcal{S},\mathcal{T})+ϵ(\forall h\in \mathcal{H})\text{\hspace{1em}(17)}$$
among ${ϵ}_{\mathcal{S}}(h,h_{\mathcal{S}}^{\ast })$ Indicates the source domain risk ,$ϵ$ Indicates adaptability ：
$$ϵ={ϵ}_{\mathcal{T}}(h_{\mathcal{T}}^{\ast })+{ϵ}_{\mathcal{S}}(h_{\mathcal{T}}^{\ast },h_{\mathcal{S}}^{\ast })$$
further , if $l$ bounded , namely $\forall (y,y^{\prime })\in {\mathcal{Y}}^2,\exists M>0$, bring $l(y,y^{\prime })\le M$. As defined $l(y,y^{\prime })=|y-y^{\prime }{|}^q$, if $\hat{\mathcal{S}}$ And $\hat{\mathcal{T}}$ Yes, the capacity is $n$ and $m$ The sample of , At least $1-\delta$ The following formula holds ：
$$d_{\mathcal{H}\Delta \mathcal{H}}(\mathcal{S},\mathcal{T})\le d_{\mathcal{H}\Delta \mathcal{H}}(\hat{\mathcal{S}},\hat{\mathcal{T}})+4q({\mathfrak{R}}_{n,\mathcal{S}}(\mathcal{H})+{\mathfrak{R}}_{m,\mathcal{T}}(\mathcal{H}))+3M\left(\sqrt{\frac{\log (4/\delta )}{2n}}+\sqrt{\frac{\log (4/\delta )}{2m}}\right)\text{\hspace{1em}(18)}$$
among ${\mathfrak{R}}_{n,\mathcal{D}}$ Express Expected radmach complexity （Expected Rademacher Complexity, reference $[6]$）.

All of the above $\mathcal{H}\Delta \mathcal{H}$ The upper bound of divergence is still too loose （ because $h$ and $h^{\prime }$ It's optional , Then the supremum value will be very large ）, So references $[204]$ Consider fixing one of these assumptions , Put forward The gap is contradictory The concept of （ Please contact the above Gap difference Distinguish , One is disparity discrepancy, One is disparity difference）：

Definition $7$（ The gap is contradictory ）：

Given Binary hypothesis space $\mathcal{H}$ And a specific assumption $h\in \mathcal{H}$, from $h$ Derived The gap is contradictory Defined as ：
$$d_{h,\mathcal{H}}(\mathcal{S},\mathcal{T})=\underset{h^{\prime }\in \mathcal{H}}{\sup }({\mathbb{E}}_{\mathcal{T}}\text{1}[h^{\prime }\ne h]-{\mathbb{E}}_{\mathcal{S}}\text{1}[h^{\prime }\ne h])\text{\hspace{1em}(19)}$$

• The authors note ：

  contrast ** Definition $4$** Medium $\mathcal{H}\Delta \mathcal{H}$ The divergence , Here is actually a fixed $h$, Nothing else makes any difference . Definitions and theorems from here down are basically of no practical significance .

At this time, the supremum is only a hypothesis $h^{\prime }$ Take office , Therefore, the upper bound is greatly reduced , And it should be easier to calculate . The gap is contradictory It can be well used to measure Distribution drift （distribution shift） The degree of .

Theorem $8$（ reference $[204]$）：

$\hat{\mathcal{S}}$ And $\hat{\mathcal{T}}$ Yes, the capacity is $n$ and $m$ The sample of , For arbitrary $\delta >0$ And every binary classifier $h\in \mathcal{H}$, At least $1-3\delta$ The following formula holds ：
$$\begin{gathered} {ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\hat{\mathcal{S}}}(\hat{\mathcal{S}},\hat{\mathcal{T}})+d_{h,\mathcal{H}}(\hat{\mathcal{S}},\hat{\mathcal{T}})+{ϵ}_{\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}}+2{\mathfrak{R}}_{n,\mathcal{S}}(\mathcal{H}) \\ +2{\mathfrak{R}}_{n,\mathcal{S}}(\mathcal{H}\Delta \mathcal{H})+2\sqrt{\frac{\log (2/\delta )}{2n}}+2{\mathfrak{R}}_{m,\mathcal{T}}(\mathcal{H}\Delta \mathcal{H})+2\sqrt{\frac{\log (2/\delta )}{2m}}\text{\hspace{1em}(20)} \end{gathered}$$

** Theorem $8$** It is the case of two categories , It can be extended to the case of multi classification , Before that, let's give a new definition ：

Definition $9$（ Marginal gap contradiction ）

Given a Scoring hypothesis space （scoring hypothesis space）$\mathcal{F}$, Make
$${\rho }_f(x,y)\stackrel{\Delta }{=}\frac{1}{2}(f(x,y)-\underset{y^{\prime }\ne y}{\max }f(x,y^{\prime }))$$
Indicates in the sample pair $(x,y)$ Situated Real hypothesis （real hypothesis）$f$ Of limit （margin）, Make
$$h_f:x\to {\text{argmax}}_{y\in \mathcal{Y}}f(x,y)$$
By $f$ Derived Tag function （labeling function）, Make
$${\Phi }_{\rho }(x)\stackrel{\Delta }{=}\{\begin{array}{rlrl}& 0& & x\ge \rho \\ & 1-\frac{x}{\rho }& & 0\le x\le \rho \\ & 1& & x\le 0\end{array}\text{\hspace{1em}(21)}$$
Express Marginal Loss （margin loss）, Is distributed $\mathcal{D}$ On ,$f$ And $f^{\prime }$ Of Marginal gap （margin disparity） by ：
$${ϵ}_{\mathcal{D}}^{(\rho )}(f^{\prime },f)={\mathbb{E}}_{(x,y)\sim \mathcal{D}}[{\Phi }_{\rho }({\rho }_{f^{\prime }}(x,h_f(x)))]\text{\hspace{1em}(22)}$$
Given specific assumptions $f\in \mathcal{F}$, be Marginal gap contradiction （margin disparity discrepancy） by ：
$$d_{f,\mathcal{F}}^{(\rho )}(\mathcal{S},\mathcal{T})=\underset{f^{\prime }\in \mathcal{F}}{\sup }[{ϵ}_{\mathcal{T}}^{(\rho )}(f^{\prime },f)-{ϵ}_{\mathcal{S}}^{(\rho )}(f^{\prime },f)]\text{\hspace{1em}(23)}$$

According to the type $(22)$ You know Marginal gap Satisfy nonnegativity and Subadditivity （subadditivity）, But it's not asymmetric , Therefore, we cannot directly ** Theorem $6$** Use this to generate a new upper bound , So we have the last theorem in this section ：

Theorem $10$（ reference $[204]$）：

stay Definition $9$ Under the assumption that , For arbitrary $\delta >0$, And whatever Scoring function $f\in \mathcal{F}$, At least $1-3\delta$ The following formula holds ：
$$\begin{gathered} {ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\hat{\mathcal{S}}}^{(\rho )}(f)+d_{f,\mathcal{F}}^{(\rho )}(\hat{\mathcal{S}},\hat{\mathcal{T}})+{ϵ}_{\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}}+\frac{2k^2}{\rho }{\mathfrak{R}}_{n,\mathcal{S}}({\Pi }_1\mathcal{F}) \\ +\frac{k}{\rho }{\mathfrak{R}}_{n,\mathcal{S}}({\Pi }_{\mathcal{H}}\mathcal{F})+2\sqrt{\frac{\log (2/\delta )}{2n}}+2{\mathfrak{R}}_{m,\mathcal{T}}({\Pi }_{\mathcal{H}}\mathcal{F})+2\sqrt{\frac{\log (2/\delta )}{2m}}\text{\hspace{1em}(24)} \end{gathered}$$

** Theorem $10$** The upper bound of the margin in indicates an appropriate margin $\rho$ It can generate better promotion results in the target field . Theorem $8$ And Theorem $10$ Co constitute Section 3.2.3 Medium Suppose the opposite （hypothesis adversarial） Method .

Notice whether it is $\mathcal{H}\Delta \mathcal{H}$ The divergence still The gap is contradictory , The supremum symbol $\sup$ Are only in hypothetical space $\mathcal{H}$ It only makes sense when you are younger , However, in the general neural network model , Hypothetical space $\mathcal{H}$ Will be very large , At this time, taking the supremum will tend to be positive and infinite and lose its meaning . However, the hypothesis space can be reduced by pre training in upstream tasks , This is it. Field opposition And Suppose the opposite Pre training necessary for the method .

---

3.2.1 Statistical matching Statistics Matching

Many upper bound theoretical results on domain adaptability have been introduced above , The problem is that most of these theories rely on Suppose the derived （hypothesis-induced） Distribution distance , These theoretical results are not very intuitive before the model is trained , Therefore, this section mainly introduces some probability results based on statistics . Be careful , reference $[112,114]$ A lot based on Suppose the derived Domain adaptive algorithm based on Distributed Distance construction .

Definition $11$（ The largest average gap ）：

Given two probability distributions $\mathcal{S},\mathcal{T}$ as well as Measurable space （measurable space）$\mathbf{X}$, Overall probability index （integral probability metric, reference $[140]$） Defined as ：
$$d_{\mathcal{F}}(\mathcal{S},\mathcal{T})\stackrel{\Delta }{=}\underset{f\in \mathcal{F}}{\sup }|{\mathbb{E}}_{\mathbf{x}\sim \mathcal{S}}[f(\mathbf{x})]-{\mathbb{E}}_{\mathbf{x}\sim \mathcal{T}}[f(\mathbf{x})]|$$
among $\mathcal{F}$ yes $\mathbf{X}$ A class of bounded functions on . reference $[163]$ Further constrain $\mathcal{F}$ To Kernel Hilbert space （kernel Hilbert space,RKHS）${\mathcal{H}}_k$ One of them Unit ball （unit ball） Inside , namely $\mathcal{F}=\{f\in {\mathcal{H}}_k:\Vert f{\Vert }_{{\mathcal{H}}_k}\le 1\}$, among $k$ yes Characteristic core （characteristic kernel）, Derived from this The largest average gap （maximum mean discrepancy,MMD, reference $[57]$）：
$$d_{\mathrm{M}\mathrm{M}\mathrm{D}}^2(\mathcal{S},\mathcal{T})=\Vert {\mathbb{E}}_{\mathbf{x}\in \mathcal{S}}[\varphi (\mathbf{x})]-{\mathbb{E}}_{\mathbf{x}\in \mathcal{T}}[\varphi (\mathbf{x})]{\Vert }_{{\mathcal{H}}_k}^2\text{\hspace{1em}(25)}$$
among $\varphi (x)$ Is with kernel function $k$ Related feature mapping , Satisfy ：
$$k(\mathbf{x},{\mathbf{x}}^{\prime })=*\varphi (\mathbf{x}),\varphi ({\mathbf{x}}^{\prime })*$$
Can prove that ,$\mathcal{S}=\mathcal{T}$ At present only if $d_{\mathcal{F}}(\mathcal{S},\mathcal{T})=0$ or $d_{\mathrm{M}\mathrm{M}\mathrm{D}}^2(\mathcal{S},\mathcal{T})=0$

Theorem $12$（ reference $[140]$）：

Given and Definition $11$ The same settings ,$l$ Is a convex loss function , Form like $l(y,y^{\prime })=|y-y^{\prime }{|}^q$, Then for any $\delta >0$ as well as $\forall h\in \mathcal{F}$, There are at least $1-\delta$ The following formula holds for the probability of ：
$$\begin{gathered} {ϵ}_{\mathcal{T}}(h)\le {ϵ}_{\mathcal{S}}(h)+d_{\mathrm{M}\mathrm{M}\mathrm{D}}(\hat{\mathcal{S}},\hat{\mathcal{T}})+{ϵ}_{\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}}+\frac{2}{n}{\mathbb{E}}_{\mathbf{x}\sim \mathcal{S}}\left[\sqrt{\text{tr}({\mathbf{K}}_{\mathcal{S}})}\right] \\ +\frac{2}{m}{\mathbb{E}}_{\mathbf{x}\sim \mathcal{T}}\left[\sqrt{\text{tr}({\mathbf{K}}_{\mathcal{T}})}\right]+2\sqrt{\frac{\log (2/\delta )}{2n}}+\sqrt{\frac{\log (2/\delta )}{2m}}\text{\hspace{1em}(26)} \end{gathered}$$
among ${\mathbf{K}}_{\mathcal{S}}$ And ${\mathbf{K}}_{\mathcal{T}}$ Respectively means according to $\mathcal{S}$ and $\mathcal{T}$ Calculated from the sample in Kernel matrix （kernel matrices）.

In fact, it's the same as the one above Gap difference It doesn't make much difference , Just redefine the new distance calculation method , And replacing assumptions with functions , But in comparison, it has the following advantages ：

1. Irrelevant to assumptions （hypothesis-free）, That is, there is no need to get an exact model to measure the distribution distance .
2. Complex items （complexity term） And Vapnik-Chervonenkis Dimension independent .
3. MMD The unbiased estimator of can be calculated in linear time .
4. MMD The process of minimization has a very beautiful statistical matching explanation in probability theory .

And MMD Relevant studies include ： reference $[174,57,58]$, What is worth noting is the references $[57,58]$ be based on Deep adaptation to the Internet （deep adaptation network,DAN, reference $[112,116]$）, Put forward MMD A variation of the Multicore MMD（multi-kernel MMD,MK-MMD）, Specific as Figure 19 Shown on the left ：

Figure 19 The middle and right pictures are references $[114]$ Proposed Joint Adaptive Network （joint adaptation network,JAN） Medium Joint maximum average gap （joint maximum mean discrepancy,JMMD）, This is used to measure two joint distributions $P({\mathbf{X}}^s,{\mathbf{Y}}^s)$ And $P({\mathbf{X}}^t,{\mathbf{Y}}^t)$ Distance between , use $\{({\mathbf{z}}_i^{s1},...,{\mathbf{z}}_i^{s|\mathcal{L}|}){\}}_{i=1}^n$ And $\{({\mathbf{z}}_i^{t1},...,{\mathbf{z}}_i^{t|\mathcal{L}|}){\}}_{j=1}^m$ Respectively represent activation and adaptation layers $\mathcal{L}$,JMMD The definition is as follows :
$$d_{\mathrm{J}\mathrm{M}\mathrm{M}\mathrm{D}}^2(\hat{\mathcal{S}},\hat{\mathcal{T}})=\Vert {\mathbb{E}}_{i\in [n]}{\otimes }_{l\in \mathcal{L}}{\varphi }^l({\mathbf{z}}_i^{sl})-{\mathbb{E}}_{j\in [m]}{\otimes }_{l\in \mathcal{L}}{\varphi }^l({\mathbf{z}}_j^{tl}){\Vert }_{{\mathcal{H}}_k}^2\text{\hspace{1em}(27)}$$
among ${\varphi }^l$ It's about kernel function $k^l$ And the network layer $l$ Feature mapping of ,$\otimes$ Represents outer product .

Commonly used in MMD The kernel function in is Gaussian kernel ：
$$k({\mathbf{x}}_1,{\mathbf{x}}_2)=\exp \left(\frac{-\Vert {\mathbf{x}}_1-{\mathbf{x}}_2{\Vert }^2}{2{\sigma }^2}\right)$$
Taylor expansion can be used to expand MMD Expressed as Statistical momentum of each order （all orders of statistic moments） Weighted sum of distances , Based on this idea , reference $[166,200]$ Yes MMD Made some similar variants .

MMD The drawback of is that it estimates the distance between two fields , The geometric information of data distribution cannot be considered , Research on the improvement of this defect includes references $[34,36,29]$

Finally, record some other relevant studies ：

• reference $[102]$： Directly by aligning Batch of standardized （BatchNorm） Statistics to minimize the domain distance , To alleviate the distribution drift problem .
• reference $[185]$： Put forward Portable standardization （transferable normalization,TransNorm）, That is to capture sufficient statistics of the field through the mean and standard deviation of the distribution of specific fields .
• reference $[84]$： Put forward Compare the adaptation network （contrastive adaptation network,CAN）, It seems to use the method of class , It is to alleviate MMD and JMMD Different categories in The samples are not complete （misalign samples） The problem of .

---

3.2.2 Domain opposite learning Domain Adversarial Learning

• Domain opposition neural network （domain adversarial neural network,DANN, reference $[45,46]$）：

  DANN Itself inspired by Against generative networks （generative adversarial net,GAN, reference $[54]$） Modeling idea of distribution . Such as Figure 20 Shown on the left ,DANN It contains two modules , The first module Domain identifier （domain discriminator）$D$ Training is used to distinguish Source characteristics （source features） And Target characteristics （target features）, The second module Feature generator （feature generator）$\psi$ Training is used to generate features to confuse $D$.

  

  according to Section 3.2 About China $\mathcal{H}\Delta \mathcal{H}$ The conclusion of divergence , It can be concluded that $\mathcal{H}\Delta \mathcal{H}$ Divergence gap ：
  $$L_{\mathrm{D}\mathrm{A}\mathrm{N}\mathrm{N}}(\psi )=\underset{D}{\max }{\mathbb{E}}_{{\mathbf{x}}^s\sim \hat{\mathcal{S}}}\log [D({\mathbf{z}}^s)]+{\mathbb{E}}_{{\mathbf{x}}^t\sim \hat{\mathcal{T}}}\log [1-D({\mathbf{z}}^t)]\text{\hspace{1em}(28)}$$
  among ,$\mathbf{z}=\psi (\mathbf{x})$ yes $\mathbf{x}$ Feature representation of , Feature generator $\psi$ The objective function of is to minimize Source error （source error） And the formula $(28)$ Medium $\mathcal{H}\Delta \mathcal{H}$ The divergence , As shown below ：
  $$\underset{\psi ,h}{\min }{\mathbb{E}}_{({\mathbf{x}}^s,{\mathbf{y}}^s)\sim \hat{\mathcal{S}}}{L}_{\text{CE}}(h({\mathbf{z}}^s),{\mathbf{y}}^s)+\lambda L(\mathrm{D}\mathrm{A}\mathrm{N}\mathrm{N})(\psi )\text{\hspace{1em}(29)}$$
  among $L_{\mathrm{C}\mathrm{E}}$ It's cross entropy loss ,$\lambda$ It's a super parameter that weighs two terms .

  reference $[115]$ fitting $(28)$ Improved , Put forward Conditional domain opposite network （conditional domain adversarial network,CDAN）, That is, the classifier predicts the result $\hat{\mathbf{y}}=h(\mathbf{z})$ Under the conditions of $\mathbf{z}$ Probability distribution of , And introduce Multilinear mapping （multilinear map）$\mathbf{z}\otimes \hat{\mathbf{y}}$ To replace $(28)$ Medium $\mathbf{z}$ As a domain identifier $D$ The input of ：
  $$L_{\mathrm{C}\mathrm{D}\mathrm{A}\mathrm{N}}(\psi )=\underset{D}{\max }{\mathbb{E}}_{{\mathbf{x}}^s\sim \hat{\mathcal{S}}}\log [D({\mathbf{z}}^s\otimes {\hat{\mathbf{y}}}^s)]+{\mathbb{E}}_{{\mathbf{x}}^t\sim \hat{\mathcal{T}}}\log [1-D({\mathbf{z}}^t\otimes {\hat{\mathbf{y}}}^t)]\text{\hspace{1em}(30)}$$
  type $(30)$ Comparative formula $(29)$ The advantage is that ,CDAN It can completely capture the relationship between feature representation and classifier prediction results Cross variance （cross-variance）, Therefore, a better joint distribution can be obtained .

• improvement ： reference $[176,28,15]$

  • reference $[176]$： Put forward Opposites distinguish areas of adaptation （adversarial discriminative domain adaptation,ADDA）, The solution is DANN The gradient disappearance problem that may appear in , To be specific ADDA The feature generator $\psi$ And domain identifier $D$ The optimization process of is divided into two independent parts , among $D$ The part and formula of $(29)$ identical ,$\psi$ Part of is transformed into ：
    $$\underset{\psi }{\min }{\mathbb{E}}_{{\mathbf{x}}^t\sim \hat{\mathcal{T}}}-\log [D({\mathbf{z}}^t)]\text{\hspace{1em}(31)}$$

  • reference $[28]$： Put forward Batch spectrum penalty （batch spectral penalization,BSP）, The feature vector corresponding to the maximum eigenvalue contains more knowledge about migration , Therefore, the penalty term about the maximum singular value is set to enhance the effect of feature classification , This is about the migration model of image processing .

  • reference $[15]$： Introduce a private subspace for each domain , It is used to save information in specific fields , This is also about the migration research of image processing .

• The application of domain opposition learning in real scenes ：

  

  • Which part should be adapted is unknown （which part to adapt is unknown）：

    Here we discuss the field of image processing . In image recognition , We only need to classify the input images , However, in entity recognition , We need to locate first Areas of interest （region of interests,RoIs） And then sort it out . Due to the distribution drift in different fields , So in the target area RoIs The location of is unreliable , Therefore, it is unknown which part of the opposite training should be adapted .

    reference $[30,147,28,82]$ Basically, the research is carried out around the solution of this problem , In fact, it is somewhat similar to the above regularization operation , It is nothing more than to align features or output results , The specific operation is as follows Figure 21 Shown .

  • There are structural dependencies between each sample tag （there are structural dependencies between labels of each sample）：

    stay Semantic division （semantic segment, This is a concept in computer vision ） And word segmentation （ Such as named entity recognition , Part of speech tagging ） And other low-level classification problems , Feature based adaptation is often not a good choice （ That is, directly embedding words as adaptation ）, The reason is that the characteristics of each pixel or word segmentation are high-dimensional , And a sample will contain a lot of pixels or word segmentation . However , There are two sides to everything , In advanced classification problems , The output space of these low-level classification tasks usually contains rich information about distribution （ Such as scene layout information or context information ）, Therefore, it is a good choice to adapt directly based on the output space .

    Relevant research includes references $[173,182]$,Figure 21 The lower right figure in corresponds to references $[173]$

---

3.2.3 Hypothetical oppositional learning Hypothesis Adversarial Learning

• reference $[146]$ Put forward Maximum classifier gap （maximum classifier discrepancy,MCD） For fully parameterized estimation and optimization $\mathcal{H}\Delta \mathcal{H}$ The divergence , Specific as Figure 22 Shown ：

  • MCD Maximize the gap between the output of two classifiers , And detect target samples that are far from the source distribution （ namely $\mathcal{H}\Delta \mathcal{H}$ The divergence ）.
  • Next, a feature generator is used to learn and generate target features to minimize the domain gap .
  • MCD It uses $L_1$ Distance to measure the gap .
  • Theoretically MCD Than $\mathcal{H}\Delta \mathcal{H}$ Divergence can get closer results , However, experiments show that its convergence speed is slow and it is very sensitive to hyperparameters . The possible reasons are MCD Two arbitrary classifiers are used in $h$ and $h^{\prime }$ To maximize the gap , bring Minimize and maximize （minimax） It is relatively difficult to reach equilibrium .

• reference $[204]$ Proposed The gap is contradictory （disparity discrepancy,DD, It seems to have been mentioned above ） Provides a tighter upper bound , The method is through the assumption space $\mathcal{H}$ Take supremum （ Instead of $\mathcal{H}\Delta \mathcal{H}$）, This significantly makes Minimize and maximize Simple . Such as Figure 22 Shown ,DD An opposite classifier is introduced $h^{\prime }$（ And $h$ Belong to the same hypothetical space ）, be $d_{h,\mathcal{H}}(\mathcal{S},\mathcal{T})$ The supremum of is approximately ：
  $$L_{\mathrm{D}\mathrm{D}}(h,\psi )=\underset{h^{\prime }}{\max }{\mathbb{E}}_{{\mathbf{x}}^s\sim \hat{\mathcal{S}}}{L}^s[h^{\prime }(\psi ({\mathbf{x}}^s)),h(\psi ({\mathbf{x}}^s))]-{\mathbb{E}}_{{\mathbf{x}}^t\sim \hat{\mathcal{T}}}{L}^s[h^{\prime }(\psi ({\mathbf{x}}^s)),h(\psi ({\mathbf{x}}^s))]\text{\hspace{1em}(32)}$$
  among $L^s$ And $L^t$ Is the loss function defined in the source domain and the target domain .

  Theoretically, the opposite classifier $h^{\prime }$ Closer to the supremum , Minimizing the following formula can reduce the target error ${ϵ}_{\mathcal{T}}$：
  $$\underset{\psi ,h}{\min }{\mathbb{E}}_{{\mathbf{x}}^s,{\mathbf{y}}^s\sim \hat{\mathcal{S}}}{L}_{\mathrm{C}\mathrm{E}}(h(\psi ({\mathbf{x}}^s)),{\mathbf{y}}^s)+\lambda L_{\mathrm{D}\mathrm{D}}(h,\psi )\text{\hspace{1em}(33)}$$
  among $\lambda$ Is the coefficient of weighing two terms . The intuitive explanation is ,DD Looking for an opposing classifier $h^{\prime }$ To correctly predict the samples in the source field , And be able to get in the target field with $h$ Different predictions , Then the feature generator $\psi$ It is trained to generate features on this decision boundary to avoid such a situation .

• DD It can only be used to deal with binary classification problems , So references $[204]$ It is further extended to Marginal gap contradiction （margin disparity discrepancy,MDD） To deal with the problem of polyphenols ：
  $$L_{\mathrm{M}\mathrm{D}\mathrm{D}}(h,\psi )=\underset{h^{\prime }}{\max }\gamma {\mathbb{E}}_{{\mathbf{x}}^s\sim \hat{\mathcal{S}}}\log [{\sigma }_{h(\psi ({\mathbf{x}}^s))}(h^{\prime }(\psi ({\mathbf{x}}^s)))]+{\mathbb{E}}_{{\mathbf{x}}^t\sim \hat{\mathcal{T}}}\log [1-{\sigma }_{h(\psi ({\mathbf{x}}^t))}(h^{\prime }(\psi ({\mathbf{x}}^t)))]\text{\hspace{1em}(34)}$$
  among $\gamma =\exp \rho$,$\rho$ Is the marginal rate ,$\sigma$ Is the activation function （ Such as softmax）,$\gamma$ The choice of is very critical , It can effectively avoid overestimating the boundary .

---

3.2.4 Domain translation Domain Translation

• Domain translation （domain translation） Refers to the original data （ Text 、 Images 、 Audio etc. ） From the source field $\mathcal{S}$ Map to the target domain $\mathcal{T}$ A kind of task in . In domain adaptability , We can use the translation model （ Usually based on GAN） Obtain tags similar to the source domain samples in the target domain .

• in the light of GAN, There are many different variants of improvements , Such as references $[106]$ Proposed Pairwise confrontation generates Networks （coupled generative adversarial networks,CoGAN） Learn the joint distribution of multi domain image data , That is, multiple different generators need to generate pairs 、 In different fields 、 Tag shared data .CoGAN Joint distribution can be learned without corresponding images in different fields , These generated target samples of shared tags will be used to train models in the target domain .

• The objective function of domain translation is usually to learn a mapping $G:\mathcal{S}\to \mathcal{T}$ Make the sample generated $G(\mathbf{x})$ It is indistinguishable from the training samples in the target field . Such as Figure 23 Shown , reference $[16]$ Put forward PixelDA Introduce an opposition identifier $D$ To distinguish the translated samples from the target samples ：
  $$L_{\mathrm{G}\mathrm{A}\mathrm{N}}(G)=\underset{D}{\max }{\mathbb{E}}_{\mathbf{x}\sim \hat{\mathcal{S}}}\log [1-D(G(\mathbf{x}))]+{\mathbb{E}}_{\mathbf{x}\sim \hat{\mathcal{T}}}\log [D(\mathbf{x})]\text{\hspace{1em}(35)}$$
  generator $G$ Try to hope to make $G(\mathbf{x})$ As similar as possible to the samples in the target field , So next we need ${\min }_G{L}_{\mathrm{G}\mathrm{A}\mathrm{N}}(G)$.

  

  Finally, the classifier of each task $f$ And feature mining $\psi$ Through supervised training （ In generating data ） adopt ：
  $$\underset{\psi ,h}{\min }{\mathbb{E}}_{(\mathbf{x},\mathbf{y})\sim \hat{\mathcal{S}}}{L}_{\sup }(h\circ \psi (G(\mathbf{x})),\mathbf{y})$$

• Periodic consistency （cycle consistency）：

  GAN It can be used to learn the mapping between data sets in different fields , Here are two questions ：

  • The sample of the source domain may be mapped to a sample independent of the target domain .
  • Samples from multiple different domains may be mapped to samples from the same target domain , This leads to classic Mode collapse （mode collapse, reference $[54]$） problem .

  From this reference $[209]$ Put forward CycleGAN, A reverse mapping is introduced $F:\mathcal{T}\to \mathcal{S}$（ Map from the target domain to the source domain ）, Build another Periodic consistency To reduce the possible mapping function space （ Such as Figure 23 Shown ）. Specifically, from the perspective of mathematical expression , Periodic consistency need $F$ and $G$ The two mappings are Double shot （bijections） And they are inverse mappings to each other , In practice $F(G(\mathbf{x}))\approx \mathbf{x},G(F(\mathbf{x}))\approx \mathbf{x}$, This idea has been widely used in domain adaptability , Such as references $[72,189,17,85,92,33]$, The last two articles are about natural language processing , It's all about cross language machine translation tasks , The first four articles are all about image processing , Including image classification 、 Semantic division 、 Identity recognition 、 Robot grabs （robotic grasping）, Entity recognition, etc .

• Semantic consistency （semantic consistency）：

  It is easy to happen when mapping labels from the source domain to the target domain Label flip （label flipping） problem , This leads to Semantic consistency It refers to the hope to make the sample $\mathbf{x}$ Consistent with the translated samples , namely $f(\mathbf{x})=f(G(\mathbf{x}))$. because $f$ Unknown , So references $[169,72,17]$ Put forward a series of Proxy functions （proxy function）, That is, in a given proxy function $h_{\mathrm{p}}$ And distance measurement $d$ Under the circumstances , The objective function is to reduce semantic inconsistencies ：
  $$\underset{G}{\min }{L}_{\mathrm{S}\mathrm{C}}(G,h_{\mathrm{p}})=d(h_{\mathrm{p}}(\mathbf{x}),h_{\mathrm{p}}(G(\mathbf{x})))\text{\hspace{1em}(36)}$$

• If the difference in the field is mainly low （ Text 、 describe 、 Color, etc. ）, Translation methods can effectively make up for domain differences , But if it is advanced （ Such as Camera angle ,camera angles）, Translation methods usually don't work .

---

3.2.5 Semi-supervised learning Semi-Supervised Learning

• Unsupervised domain adaptability （UDA） And Semi-supervised learning （SSL） Is closely related to the , many SSL Methods can be applied to UDA in .

• Semi-supervised learning Three assumptions ：

  • Smoothness assumption （smoothness assumption）： If two samples ${\mathbf{x}}_1,{\mathbf{x}}_2$ In high-density areas, the distance is similar , Then their output ${\mathbf{y}}_1,{\mathbf{y}}_2$ It should also be similar .
  • Cluster hypothesis （cluster assumption）： If the sample points are in the same cluster , Then they are probably of the same type .
  • manifold hypothesis （ assumption）： High dimensional data can be approximated to a low dimension manifold（ This word seems to be untranslatable ） in .

• Can be used for UDA Some of SSL Method ：

  • Uniform regularization （consistency regularization）： It means that similar sample points should get consistent prediction results , Relevant research includes references $[44,48]$.

  • Entropy minimization （entropy minimization）： Make the model have the highest confidence for the unlabeled data （ That is, the lowest entropy ） Predicted results , It is often used as an auxiliary method of adaptive methods in many fields , Relevant research includes references $[82.104]$.

  • False labeling （pseudo-labeling）： Generate proxy labels for unmarked data , And use these noisy tags to mix with the labeled data for training . Relevant research includes references $[212,201,205]$, It is particularly mentioned here that the cross entropy loss is highly sensitive to noise labels , So references $[143,105]$ It is proposed that Generalized cross entropy loss （generalized cross-entropy,GCE） As a solution ：
    $$L_{\mathrm{G}\mathrm{C}\mathrm{E}}(\mathbf{x},\tilde{y})=\frac{1}{q}(1-h_{\tilde{y}}(\mathbf{x}{)}^q)\text{\hspace{1em}(37)}$$
    among $q\in (0,1]$ It is a super parameter used to weigh the two loss functions of cross entropy and mean absolute value error .

---

3.2.6 notes Remarks

Summarize the performance of all methods in this section ：

Method	Adaptability	Data efficiency	Modal ductility	Task extensibility	Theoretical guarantee
Statistical matching	Bad	optimal	optimal	in	optimal
Domain opposite learning	in	in	optimal	in	optimal
Hypothetical oppositional learning	optimal	in	optimal	in	optimal
Domain translation	in	Bad	Bad	optimal	Bad
Semi-supervised learning	in	in	in	Bad	Bad

Field description ：

• Adaptability （adaptation performance）： Model performance when there is a large amount of annotation data in downstream tasks .
• Data efficiency （data efficiency）： Model performance when there is only a small amount of data in downstream tasks .
• Modal ductility （modality scalability）： Can it be used for multimodal data , Text 、 picture 、 Audio-visual .
• Task extensibility （task scalability）： Can you easily migrate the pre training model to different downstream tasks .
• Theoretical guarantee （theory guarantee）： Whether the generalization error of the target field in adaptation can be controlled within a certain limit .

---

4 assessment Evaluation

4.1 Data sets Datasets

General language comprehension assessment （general language understanding evaluation,GLUE, reference $[183]$） It is the most famous benchmark in the field of natural language processing , The following table illustrates a series of GLUE Data set of , Language comprehension tasks including nine sentences or sentence pairs ：

Corpus	The amount of training data	Test data volume	Evaluation indicators	Task type	field
$\text{CoLA}$	$8.5\mathrm{k}$	$1\mathrm{k}$	Matthews correlation coefficient	Acceptability	blend
$\text{SST-2}$	$67\mathrm{k}$	$1.8\mathrm{k}$	accuracy	Sentiment analysis	Film review
$\text{MRPC}$	$3.7\mathrm{k}$	$1.7\mathrm{k}$	accuracy /$\text{F1-score}$	The phrase	Journalism
$\text{STS-B}$	$7\mathrm{k}$	$1.4\mathrm{k}$	Pearson correlation coefficient	Sentence similarity	blend
$\text{QQP}$	$364\mathrm{k}$	$391\mathrm{k}$	accuracy /$\text{F1-score}$	The phrase	Social Q & A
$\text{MNLI}$	$393\mathrm{k}$	$20\mathrm{k}$	（ Not ） Matching accuracy	Natural language inference	blend
$\text{QNLI}$	$105\mathrm{k}$	$5.4\mathrm{k}$	accuracy	Question and answer / Natural language inference	Wikipedia
$\text{RTE}$	$2.5\mathrm{k}$	$3\mathrm{k}$	accuracy	Natural language inference	News and Wikipedia
$\text{WNLI}$	$634$	$146$	accuracy	Co refer to / Natural language inference	Science fiction books

• The authors note ： The last data set may have a small amount of data .

But there is still no similar GLUE Benchmark for computer vision , Here are just some commonly used data sets in the field of image processing ：

Data sets	The amount of training data	Test data volume	Number of categories	Evaluation indicators	field
$\text{Food-101}$（ reference $[88]$）	$75750$	$25250$	$101$	$\text{top-1}$	blend
$\text{CIFAR-10}$（ reference $[88]$）	$50000$	$10000$	$10$	$\text{top-1}$	blend
$\text{cIFAR-100}$（ reference $[88]$）	$50000$	$10000$	$100$	$\text{top-1}$	blend
$\text{SUN397}$（ reference $[88]$）	$19850$	$19850$	$397$	$\text{top-1}$	blend
$\text{Stanford Cars}$（ reference $[88]$）	$8144$	$8041$	$196$	$\text{top-1}$	blend
$\text{FGVC Aircraft}$（ reference $[88]$）	$6667$	$3333$	$100$	$\text{mean per-class}$	blend
$\text{Describable Textures(DTD)}$（ reference $[88]$）	$3760$	$1880$	$47$	$\text{top-1}$	blend
$\text{Oxford-III Pets}$（ reference $[88]$）	$3680$	$3369$	$37$	$\text{mean per-class}$	blend
$\text{Caltech-101}$（ reference $[88]$）	$3060$	$6084$	$102$	$\text{mean per-class}$	blend
$\text{Oxford 102 flowers}$	$2040$	$6149$	$102$	$\text{top-1}$	blend
$\text{ImageNet-R}$（ reference $[69]$）	$-$	$30\mathrm{k}$	$200$	$\text{top-1}$	blend
$\text{ImageNet-Sketch}$（ reference $[84]$）	$-$	$50\mathrm{k}$	$1000$	$\text{top-1}$	draft
$\text{DomainNet-c}$（ reference $[128]$）	$33525$	$14604$	$365$	$\text{top-1}$	Clip art
$\text{DomainNet-p}$（ reference $[128]$）	$50416$	$21850$	$365$	$\text{top-1}$	Oil Painting
$\text{DomainNet-r}$（ reference $[128]$）	$120906$	$52041$	$365$	$\text{top-1}$	blend
$\text{DomainNet-s}$（ reference $[128]$）	$48212$	$20916$	$365$	$\text{top-1}$	draft

---

4.2 Open source package Library

• Project address ：[email protected]（ Have in hand ）

• Usage method ： Official documents @TLlib

  Here's a simple one DANN Application code example ：

  # define the domain discriminator
  from dalib.modules.domain_discriminator import DomainDiscriminator
  discriminator = DomainDiscriminator(in_feature=1024, hidden_size=1024)
  
  # define the domain adversarial loss module
  from dalib.adptation.dann import DomainAdversarialLoss
  dann = DomainAdversarialLoss(discriminator, reduction='mean')
  
  # features from the source and target domain
  f_s, f_t = torch.randn(20, 1024), torch.randn(20, 1024)
  
  # calculate the final loss
  loss = dann(f_s, f_t)
  

• at present TLlib mainly PyTorch Implementation as a backend , It has good ductility .

---

4.3 The benchmark Benchmark

This section is mainly about Section 4.1 The benchmark of typical methods of pre training and adaptation on several large-scale data sets mentioned in , Some of the benchmark results are through TLlib Realize the .

4.3.1 Preliminary training Pre-Training

---

4.3.2 Task adaptability Task Adaptation

---

4.3.3 Domain adaptability Domain Adaptation

---

5 Conclusion Conclusion

In this paper, we investigate how to acquire and apply transferability in the whole lifecycle of deep learning. In the pre-training section, we focus on how to improve the transferability of the pre-trained models by designing architecture, pre-training task, and training strategy. In the task adaptation section, we discuss how to better preserve and utilize the transferable knowledge to improve the performance of target tasks. In the domain adaptation section, we illustrate how to bridge the domain gap to increase the transferability for real applications. This survey connects many isolated areas with their relation to transferability and provides a unified perspective to explore transferability in deep learning. We expect this study will attract the community’s attention to the fundamental role of transferability in deep learning.

reference

[001] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In ICLR, 2022. 
[002] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In ACL, 2021. 
[003] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In ICML, 2016. 
[004] Martin Arjovsky, L´eon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. 
[005] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In ICML, 2017. 
[006] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. In JMLR, 2002. 
[007] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: Pretrained language model for scientific text. In EMNLP, 2019. 
[008] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79, page 151–175, 2010a. 
[009] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurIPS, 2006. 
[010] Shai Ben-David, Tyler Lu, Teresa Luu, and David Pal. Impossibility theorems for domain adaptation. In AISTATS, pages 129–136, 2010b. 
[011] Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In ICML workshop, 2012. 
[012] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In NeurIPS, 2007. 
[013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 35(8):1798–1828, 2013. 
[014] Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Communications of the ACM, 64(7):58–65, 2021. 
[015] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurIPS, 2016. 
[016] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017. 
[017] Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In ICRA, 2018. 
[018] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. 
[019] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In ICCV, 2017. 
[020] Rich Caruana. Multitask learning. Technical report, 1997. 
[021] Olivier Chapelle, Bernhard Sch¨olkopf, and Alexander Zien. Semi-Supervised Learning (Adaptive Computation and Machine Learning). The MIT Press, 2006. ISBN 0262033585. 
[022] Minmin Chen, Zhixiang Xu, Kilian Q. Weinberger, and Fei Sha. Marginalized denoising autoencoders for domain adaptation. In ICML, 2012. 
[023] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 
[024] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2019a. 
[025] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021. 
[026] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021a. 
[027] Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NeurIPS, 2019b. 
[028] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In ICML, 2019c. 
[029] Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long. Representation subspace distance for domain adaptation regression. In ICML, 2021b. 
[030] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster R-CNN for object detection in the wild. In CVPR, 2018. 
[031] Kyunghyun Cho, Bart van Merri¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In EMNLP, 2014. 
[032] Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. An embarrassingly simple approach for transfer learning from pretrained language models. In NAACL, 2019. 
[033] Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In EMNLP, 2018. 
[034] Nicolas Courty, R´emi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In NeurIPS, 2017. 
[035] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale finegrained categorization and domain-specific transfer learning. In CVPR, pages 4109–4118, 2018. 
[036] Bharath Bhushan Damodaran, Benjamin Kellenberger, R´emi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In ECCV, 2018. 
[037] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI, page 1–20, 2021. 
[038] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In CVPR, 2009. 
[039] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 
[040] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. 
[041] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014. 
[042] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021. 
[043] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017. 
[044] Geoffrey French, Michal Mackiewicz, and Mark H. Fisher. Self-ensembling for domain adaptation. In ICLR, 2018. 
[045] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015. 
[046] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Fran¸cois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(59):1–35, 2016. 
[047] Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In ICLR, 2018. 
[048] Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification. In ICLR, 2020. 
[049] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019. 
[050] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 
[051] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, 2011. 
[052] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, 2012. 
[053] Boqing Gong, Kristen Grauman, and Fei Sha. Connecting the dots with landmarks: Discriminatively learning domain-invariant features for unsupervised domain adaptation. In ICML, 2013. 
[054] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NeurIPS, 2014. 
[055] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2015. 
[056] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Sch¨olkopf. Recurrent independent mechanisms. In ICLR, 2021. 
[057] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch¨olkopf, and Alexander Smola. A kernel two-sample test. JMLR, 13(25):723–773, 2012a. 
[058] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In NeurIPS, 2012b. 
[059] Jean-Bastien Grill, Florian Strub, Florent Altch´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent - a new approach to self-supervised learning. In NeurIPS, 2020. 
[060] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021. 
[061] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL, 2021. 
[062] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In CVPR, 2019. 
[063] Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, 2020. 
[064] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 
[065] Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 
[066] Kaiming He, Ross Girshick, and Piotr Doll´ar. Rethinking imagenet pre-training. In ICCV, 2019. 
[067] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 
[068] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021. 
[069] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-ofdistribution generalization. ICCV, 2021. 
[070] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019. 
[071] Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation. 2016. 
[072] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018. 
[073] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In ICML, 2019. 
[074] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL, 2018. 
[075] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Pre-training graph neural networks. In ICLR, 2020. 
[076] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Sch¨olkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In NeurIPS, 2007. 
[077] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 
[078] Yunhun Jang, Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Learning what and where to transfer. In ICML, 2019. 
[079] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In ACL, 2020. 
[080] Junguang Jiang, Yifei Ji, Ximei Wang, Yufeng Liu, Jianmin Wang, and Mingsheng Long. Regressive domain adaptation for unsupervised keypoint detection. In CVPR, 2021. 
[081] Junguang Jiang, Baixu Chen, Jianmin Wang, and Mingsheng Long. Decoupled adaptation for cross-domain object detection. In ICLR, 2022. 
[082] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In ECCV, 2020. 
[083] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. In TACL, 2020. 
[084] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019. 
[085] Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, and Changick Kim. Diversify and match: A domain adaptive representation learning paradigm for object detection. In CVPR, 2019. 
[086] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka GrabskaBarwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 114 (13):3521–3526, 2017. 
[087] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020. 
[088] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In CVPR, 2019. 
[089] Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. In NeurIPS, 2020. 
[090] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012. 
[091] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In NeurIPS, 2019. 
[092] Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2017. 
[093] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. 
[094] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): 436–444, 2015. 
[095] Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, 2019. 
[096] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020a. 
[097] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020b. 
[098] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-toSequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL, 2020. 
[099] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018. 
[100] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021. 
[101] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2019. 
[102] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In ICLR Workshop, 2017. 
[103] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):2935–2947, 2018. 
[104] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020. 
[105] Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NeurIPS, 2021a. 
[106] Ming-Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In NeurIPS, 2016. 
[107] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021b. 
[108] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019a. 
[109] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019b. 
[110] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. 2019c. 
[111] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In ICCV, 2013. 
[112] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015. 
[113] Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Unsupervised domain adaptation with residual transfer networks. In NeurIPS, 2016. 
[114] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017. 
[115] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurIPS, 2018. 
[116] Mingsheng Long, Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Transferable representation learning with deep adaptation networks. TPAMI, 41(12):3071–3085, 2019. 
[117] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l 0 regularization. In ICLR, 2018. 
[118] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 
[119] Arun Mallya and Svetlana Lazebnik. Piggyback: Adding multiple tasks to a single, fixed network by learning to mask. In ECCV, 2018. 
[120] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009. 
[121] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017. 
[122] Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07056, 2018. 
[123] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In ICML, 2020. 
[124] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. NeurIPS, 2019. 
[125] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, pages 1345–1359, 2010. 
[126] Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. TNNLS, pages 199–210, 2011. 
[127] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 
[128] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019. 
[129] Jonas Peters, Peter B¨uhlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pages 947–1012, 2016. 
[130] Jonas Peters, Dominik Janzing, and Bernhard Sch¨olkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017. 
[131] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018. 
[132] Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? In ACL, 2019. 
[133] J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. The MIT Press, 2009. 
[134] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018. 
[135] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 
[136] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020. 
[137] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In ICLR, 2020. 
[138] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In NeurIPS, 2019. 
[139] S-A Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In NeurIPS, 2017. 
[140] Ievgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Youn`es Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees, 2020. 
[141] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015. 
[142] Michael T. Rosenstein. To transfer or not to transfer. In NeurIPS, 2005. 
[143] Evgenia. Rusak, Steffen Schneider, Peter Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Adapting imagenet-scale models to complex distribution shifts with self-learning. arXiv preprint arXiv:2104.12928, 2021. 
[144] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211– 252, 2015. 
[145] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019. 
[146] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018. 
[147] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution alignment for adaptive object detection. In CVPR, 2019. 
[148] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NeurIPS, 2020. 
[149] Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, 2018. 
[150] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In ICML, 2016. 
[151] Timo Schick and Hinrich Sch¨utze. Exploiting cloze questions for few-shot text classification and natural language inference. In EACL, 2020. 
[152] J¨urgen Schmidhuber. Evolutionary principles in self-referential learning. PhD thesis, Technische Universit¨at M¨unchen, 1987. 
[153] Bernhard Sch¨olkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In ICML, 2012. 
[154] Bernhard Sch¨olkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021. 
[155] Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin ˇZ´ıdek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577 (7792):706–710, 2020. 
[156] Pierre Sermanet, David Eigen, Xiang Zhang, Micha¨el Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. 
[157] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In CVPR, 2017. 
[158] Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In ICLR, 2018. 
[159] Yang Shu, Zhangjie Cao, Jinghan Gao, Jianmin Wang, and Mingsheng Long. Omni-training for data-efficient deep learning. arXiv preprint arXiv:2110.07510, 2021a. 
[160] Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from a zoo of models. In ICML, 2021b. 
[161] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. 
[162] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017. 
[163] Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Sch¨olkopf, and Gert R. G. Lanckriet. Hilbert space embeddings and metrics on probability measures. JMLR, 2010. 
[164] Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert M¨uller. Covariate shift adaptation by importance weighted cross validation. JMLR, 8(35):985–1005, 2007. 
[165] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In NeurIPS, 2008. 
[166] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016. 
[167] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, 2019a. 
[168] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019b. 
[169] Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. In ICLR, 2017. 
[170] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998. 
[171] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020. 
[172] Lisa Torrey and Jude Shavlik. Transfer learning. 2010. 
[173] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018. 
[174] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. 2014. 
[175] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In ICCV, pages 4068–4076, 2015. 
[176] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017. 
[177] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, �Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 
[178] Petar Veliˇckovi´c, William Fedus, William L Hamilton, Pietro Li`o, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019. 
[179] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008. 
[180] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurIPS, 2016. 
[181] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha¨el Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350–354, 2019. 
[182] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Perez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019. 
[183] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019a. 
[184] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019b. 
[185] Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Transferable normalization: Towards improving transferability of deep neural networks. In NeurIPS, 2019c. 
[186] Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for dataefficient deep learning. In ICML, 2021. 
[187] Zirui Wang, Zihang Dai, Barnab´as P´oczos, and Jaime G. Carbonell. Characterizing and avoiding negative transfer. In CVPR, 2019d. 
[188] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022. 
[189] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In CVPR, 2018. 
[190] Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019. 
[191] Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018. 
[192] Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 
[193] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. In EMNLP, 2021. 
[194] I Zeki Yalniz, Herv´e J´egou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546, 2019. 
[195] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019. 
[196] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured metalearning. In ICML, 2019. 
[197] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014. 
[198] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021. 
[199] Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 
[200] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschl¨ager, and Susanne Saminger-Platz. Central moment discrepancy (cmd) for domain-invariant representation learning. In ICLR, 2017. 
[201] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017. 
[202] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019a. 
[203] Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: Network adaptation via additive side networks. 2019b. 
[204] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML, 2019c. 
[205] Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018. 
[206] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In ICLR, 2021. 
[207] Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset. In ICAIL, 2021. 
[208] Jincheng Zhong, Ximei Wang, Zhi Kou, Jianmin Wang, and Mingsheng Long. Bi-tuning of pre-trained representations. arXiv preprint arXiv:2011.06182, 2020. 
[209] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017. 
[210] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015. 
[211] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2021. 
[212] Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018.