An Evaluation of the Accuracy of the Machine Translation Systems of Social Media Language: Google’s Arabic-English Translation as an Example

Abstract

In this age of information technology, it has become possible for people all over the world to communicate in different languages through social media platforms with the help of machine translation (MT) systems. As far as the Arabic-English language pair is concerned, most studies have been conducted on evaluating the MT output for the standard varieties of Arabic, with fewer studies focusing on the vernacular or colloquial varieties. This study attempts to address this gap through presenting an evaluation of the performance of MT output for vernacular or colloquial Arabic in the social media domain. As it is currently the most widely used MT system, Google Translate (GT) has been chosen for evaluating the reliability of its output in the context of translating the Arabic colloquial language (i.e., Egyptian/Cairene Arabic variety) used in social media into English. With this goal in mind, a corpus consisting of Egyptian dialectal Arabic sentences were collected from social media networks, i.e., Facebook and Twitter, and then fed into GT system. The GT output was then evaluated by three human translators to assess their accuracy of translation in terms of adequacy and fluency. The results of the study show that several translation problems have been spotted for GT output. These problems are mainly concerned with wrong equivalents, inappropriate additions and deletions, and transliteration for out-of-vocabulary (OOV) words, which are mostly due to the literal translation of the Arabic vernacular sentences into English. This can be due to the fact that Arabic vernacular varieties are different from the standard language for which MT systems have been basically developed. This, consequently, necessitates the need to upgrade such MT systems to deal with the vernacular varieties.