LEMMA-ROUGE: An Evaluation Metric for Arabic Abstractive Text Summarization

Abstract

High morphological languages are characterized by complex inflections and derivations, which can present challenges for natural language processing tasks such as summarization. Abstractive text summarization aims to generate a summary by understanding the meaning of the text, rather than solely relying on the words used in the original source. However, few works address the  generation of abstractive summaries due to its complexity. One of the challenges is the absence of a reliable metric to evaluate the performance of abstractive summaries. This paper proposes a lemma-based ROUGE metric and investigates the effectiveness of normalization forms in the similarity matching of the ROUGE metric for evaluating abstractive text summarization systems. We use Arabic as a case study and compare results involving different forms of the word: as is, stem-based, and lemma-based. The results show that the lemma-based form achieves higher ROUGE scores than the other forms. The findings emphasize the impact of morphological complexity on the performance of abstractive text summarization systems.