Transfer learning using progressive neural networks and NMT for classification tasks in NLP

Abstract

Recently neural networks are obtaining state of the art results on many NLP tasks like sentiment classification, machine translation, etc. However one of the drawbacks of these techniques is that they need large amounts of training data. Even though there is a lot of data being generated everyday, not all tasks have large amounts of data. One possible solution when data is not sufficient is using transfer learning techniques. In this paper, we explored methods of transfer learning (or sharing the parameters) between different tasks so that the performance on the low data resource tasks is improved. We have first tried to replicate the prior results of transfer learning in semantically related tasks. When we have semantically different tasks, we tried using Progressive Neural Networks. We also experimented on sharing the encoder from neural machine translator to classification tasks.