Metric for Automatic Machine Translation Evaluation based on Pre-trained Sentence Embeddings

Abstract

This study describes a segment-level metric for automatic machine translation evaluation (MTE). Although various MTE metrics have been proposed, most MTE metrics, including the current de facto standard BLEU, can handle only limited information for segment-level MTE. Therefore, we propose an MTE metric using pre-trained sentence embeddings in order to evaluate MT translation considering global information. In our proposed method, we obtain sentence embeddings of MT translation and reference translation using a sentence encoder pre-trained on a large corpus. Then, we estimate the translation quality by a regression model based on sentence embeddings of MT translation and reference translation as input. Our metric achieved state-of-the-art performance in segment-level metrics tasks for all to-English language pairs on the WMT dataset with human evaluation score.