Macular: A Multi-Task Adversarial Framework for Cross-Lingual Natural Language Understanding

Abstract

Cross-lingual natural language understanding∼(NLU) aims to train NLU models on a source language and apply the models to NLU tasks in target languages, and is a fundamental task for many cross-language applications. Most of the existing cross-lingual NLU models assume the existence of parallel corpora so that words and sentences in source and target languages could be aligned. However, the construction of such parallel corpora is expensive and sometimes infeasible. Motivated by this challenge, recent works propose data augmentation or adversarial training methods to reduce the reliance on external parallel corpora. In this paper, we propose an orthogonal and novel perspective to tackle this challenging cross-lingual NLU task (i.e., when parallel corpora are unavailable). We propose to conduct multi-task learning across different tasks for mutual performance improvement on both source and target languages. The proposed multi-task learning framework is complementary to existing studies and could be integrated with existing methods to further improve their performance on challenging cross-lingual NLU tasks. Towards this end, we propose a multi-task adversarial framework for cross-lingual NLU, namely Macular. The proposed Macular includes a multi-task module and a task-specific module to infer both the common knowledge across tasks and unique task characteristics. More specifically, in the multi-task module, we incorporate a task adversarial loss into training to ensure the derivation of task-shared knowledge only by the representations. In the task-specific fine-tuning module, we extract task-specific knowledge which is not captured by the multi-task module. A task-level consistency loss is added to the training loss so that consistent predictions across a target task and an auxiliary task (i.e., the task that is the most similar to the target task) are achieved. A language adversarial loss is also incorporated so that knowledge can be transferred from source languages to target ones. To validate the effectiveness of the proposed Macular, we conduct extensive experiments on four public datasets including paraphrase identification, natural language understanding, question answering matching, and query advertisement matching. The experimental results show that the proposed Macular can outperform state-of-the-art cross-lingual NLU approaches.