Exploring multilingual syntactic sentence representations

Abstract

We study methods for learning sentence embeddings with syntactic structure. We focus on methods of learning syntactic sentenceembeddings by using a multilingual parallelcorpus augmented by Universal Parts-of- Speech tags. We evaluate the quality of the learned embeddings by examining sentencelevel nearest neighbours and functional dissimilarity in the embedding space. We also evaluate the ability of the method to learn syntactic sentence-embeddings for low-resource languages and demonstrate strong evidence for transfer learning. Our results show that syntactic sentence-embeddings can be learned while using less training data, fewer model parameters, and resulting in better evaluation metrics than state-of-the-art language models.