Domain adaptation via identical distribution across models and tasks

Abstract

Deep convolution neural network (CNN) models with millions of parameters trained in large-scale datasets make domain adaptation difficult to be realized. In order to be applied for different application scenarios, various light weight network models have been proposed. These models perform well in large-scale datasets but are hard to train from randomly initialized weights when lack of data. Our framework is proposed to connect a pre-trained deep model with a light weight model by enforcing feature distributions of the two models being identical. It is proved in our work that knowledge in source model can be transferred to target light weight model by identical distribution loss. Meanwhile, distribution loss allows training dataset to utilize sparse labeled data in semi-supervised classification task. Moreover, distribution loss can be applied to large amount of unlabeled data from target domain. In the experiments, several standard benchmarks on domain adaptation are evaluated and our work gets state-of-the-art performance.