DDTL: Domain Transfer Learning at a Distance

DDTL问题：The target domain can be completely different from the source domain.

1、A selective learning algorithm is proposed（SLA）,以Supervised autoencoder or supervised convolutional autoencoderAs a base model for handling different types of input.

2、SLAThe algorithm gradually selects from the intermediate domainUseful unlabeled data作为桥梁,in order to break down the huge distributional differences in the transfer of knowledge between two distant domains.

迁移学习：Learning methods that borrow knowledge from the source domain to enhance the learning ability of the target domain

Task1：Transfer knowledge between cat and tiger images.Transfer learning algorithms achieve better performance than some supervised learning algorithms.

Task2：Transfer knowledge between human faces and airplane images.The transfer learning algorithm failed,Because its performance is worse than supervised learning algorithms.然而,当应用SLA算法时,获得了更好的性能.

1、DDTL问题定义

大小为的Source field tag data：;

大小为的Target domain tag data：;

多个The intermediate fields are unlabeled data的混合:.

A domain corresponds to one of a particular classification problemconcept or category,Such as recognizing faces or airplanes from images.

问题描述：Assume that the classification problems in both the source and target domains are binary.All data points should be in the same feature space.设、和are the source domain data边际分布、Conditional and joint distributions;The three distributions relative to the target domain are ,,;is the marginal distribution of the intermediate domain.在DDTL问题中：

;

;

.

目标：利用中间域中的未标记数据,each other in the original距离较远Build a bridge between the source and target domains,And through the bridge from the source domainTransfer supervision knowledge来训练目标域accurate classifier.

PS：Not all data in the intermediate domain should be similar to the source domain data,Some of these data can be very different.因此,简单地Building bridges using all intermediate data may fail.

2、SLA：Selective Learning Algorithms 

2.1Autoencoders and their variants

       自动编码器是一种Unsupervised Feedforward Neural Networks,具有输入层、One or more hidden and output layers,It usually consists of two processes：编码和解码.

  输入：;

编码函数：Encode it to map it to a hidden representation;

解码函数：Decode to reconstructx.

The process of the autoencoder can be summarized as：

编码：;

解码：.

其中is an approximation to the original inputxthe reconstructed input.By minimizing the reconstruction error on all training data,即

2.2Instance selection by reconstruction error

        在实践中,Because the source domain and the target domain are far apart,Only a portion of the source domain data may be useful to the target domain,The situation is similar for intermediate domains.因此,In order to select useful instances from the intermediate domain,And remove the relevant instance of the target domain from the source domain,通过Minimize the reconstruction error for selected instances in the source and intermediate domains and all instances in the target domainto learn a pair of encoding and decoding functions.The objective function to be minimized is formulated as follows：

、∈ ｛0,1｝：source domainithe first instance and the intermediate domainj个实例selection indicator.当值为1时,The corresponding instance will be selected,Otherwise it will be deselected.

：和Regularization term on ,通过将和All values ​​of are set to zero to avoid some unimportant solutions.将其定义为：

        Minimizing this term is equivalent to encouraging from the source domain and the intermediate domainChoose as many instances as possible.Two regularization parameters和Controls the importance of this regularization term.

2.3Explanation of auxiliary information

Incorporate auxiliary information when learning hidden representations for different domains.

源域和目标域：Data tags can be used as auxiliary information;

中间域：没有标签信息.

       Treat predictions on intermediate domains as auxiliary information,And use the predicted confidence to guide the learning of hidden representations.具体而言,We propose to incorporate auxiliary information into learning by minimizing the following function：

：is the classification function that outputs the classification probability;

g（·）：定义为 ,其中0≤ z≤ 1;

将用于选择Instances of high prediction confidence in the intermediate domain.

2.4总体目标函数

DDTL的最终目标函数如下：

其中;Θ表示函数、和的所有参数.

Use block coordinatesdecedent（BCD）方法,在每次迭代中,在保持其他变量不变的情况下,Sequentially optimize the variables in each block.

 在selected to have low reconstruction error and low training loss;在selected to have low reconstruction error and low training loss.

The deep learning architecture is shown in the figure below.

 3、总结

         This paper studies a new oneDDTL问题,The source domain and the target domain are far apart,But can be connected through some intermediate domains.为了解决DDTL问题,提出了SLA算法,from the intermediate domainGradually select unlabeled data,to connect two distant domains.