Cross-lingual transfer learning with data selection for large-scale spoken language understanding

Abstract

A typical cross-lingual transfer learning approach boosting model performance on a resource-poor language is to pre-train the model on all available supervised data from another resource-rich language. However, in large-scale systems, this leads to high training times and computational requirements. In addition, characteristic differences between the source and target languages raise a natural question of whether source-language data selection can improve the knowledge transfer. In this paper, we address this question and propose a simple but effective language model based source-language data selection method for cross-lingual transfer learning in large-scale spoken language understanding. The experimental results show that with data selection i) the source data amount and hence training speed is reduced significantly and ii) model performance is improved.