Assessing the Quality of AI and MT in Legal Translation

Abstract

Artificial Intelligence (AI) has made huge strides forward in the last few years and many AI-driven solutions have been developed to cater for the needs of legal professionals. Amongst its various applications, AI can be used to draft and translate documents. This paper wishes to assess the accuracy and reliability of the translation (from Italian into English) of an informative article on mortgage contracts. To do so, an MT-based platform (DeepL) and two generic AI-driven chatbots (i.e., ChatGPT and Copilot) are considered. In addition, the AI-powered Notebook LM is also addressed as it can provide more targeted and field-related feedback. The paper findings highlight that, although the MT- and AI-based interfaces do not produce inconsistencies, grammatical errors, word order issues or oddities of any kind, their outputs are not satisfactory. All the automated translations, in fact, lack references to system-bound terminology and system-specific legal principles. The MT and AI systems, in fact, are not capable of disambiguating terms (especially polysemous words) in context. Therefore, at the time being, MT and AI platforms still have considerable progress to make before being fully reliable and accurate in the legal translation domain.