Metamorphic Robustness Testing for DeepL Translation

Abstract

Due to the complexity and flexibility of natural languages, such robustness testing involving the ability of machine translation software to handle incorrect or unintended input are extremely challenging, especially if there is no human oracle or reference translation. In this paper, using metamorphic robustness testing methods, we compare the translation of the primitive input with the translation of a subsequent input that has different types of very small changes and may cause large changes in the translation results. The lack of robustness of the DeepL translation is shown by our experiments, these preliminary results show that metamorphic testing has potential in the field of natural language processing.