Error Analysis of Pretrained Language Models (PLMs) in English-to-Arabic Machine Translation

Abstract

Advances in neural machine translation utilizing pretrained language models (PLMs) have shown promise in improving the translation quality between diverse languages. However, translation from English to languages with complex morphology, such as Arabic, remains challenging. This study investigated the prevailing error patterns of state-of-the-art PLMs when translating from English to Arabic across different text domains. Through empirical analysis using automatic metrics (chrF, BERTScore, COMET) and manual evaluation with the Multidimensional Quality Metrics (MQM) framework, we compared Google Translate and five PLMs (Helsinki, Marefa, Facebook, GPT-3.5-turbo, and GPT-4). Key findings provide valuable insights into current PLM limitations in handling aspects of Arabic grammar and vocabulary while also informing future improvements for advancing English–Arabic machine translation capabilities and accessibility.