Enhancing the assessment of (Polish) translation in PROMIS using statistical, semantic, and neural network metrics

Abstract

Differences in culture and language create the need for translators to convert text from one language into another. In order to preserve meaning, context must be analyzed in detail in translation. This study aims to develop accurate evaluation metrics for translations within the PROMIS (Patient-Reported Outcomes Measurement Information System) process, particularly in the reconciliation step, by providing reviews by experts as additional information following backward translation. The result is a semi-automatic semantic evaluation metric for Polish based on the concept of the human-aided translation evaluation metric (HMEANT). We assessed the proposed metrics using a statistics-based support vector machine classifier and applied deep neural networks to replicate the operation of the human brain. We compared the results of the proposed metrics with human judgment and well-known machine translation metrics, such as BLEU (Bilingual Evaluation Understudy), NIST, TER (Translation Error Rate), and METEOR (Metric for Evaluation of Translation with Explicit Ordering). We found that a few of the proposed metrics were highly correlated with human judgment and provided additional semantic information independent of human experience. This showed that the proposed metrics can help assess translations in PROMIS.