Long Document Summarization with Transformer Models: A Comparative Evaluation on the Gutenberg Dataset

Abstract

The automatic text summarization is one of the core tasks in the area of natural language processing (NLP), allowing the processing of larger amounts of information. While great progress has been made in automatic text summarization, the problem of text summarization of long documents still remains due to challenges such as content redundancy, loss of context relations, and input disability of the traditional Transformer models. Through this study, we are concerned with assessing and comparatively discussing the performance of three Transformer-based models, named BART, T5, and Flan-T5, with respect to long document summarization. Their advantages and disadvantages will be spotlighted. The experimental setup consists of using Project Gutenberg, one of the largest online libraries, as our dataset. The main long-text specimen is Alice’s Adventures in Wonderland. We evaluate the models and their variations against ROUGE and BERTScore metrics, which determine how well a summary retains the content and the level of semantic similarity between generated summaries and original texts. With the BART model, we get the best ROUGE-L F1 score of 0.0221, emphasizing the model’s effective extractive ability. T5, on the other hand, gets a ROUGE-1 recall score of 0.5000, indicating that it covers more contents across documents. This paper harvests total benchmark results of long document summarization using Transformer-based models, leaving the authors with the idea that there are countervailed performance trade-offs among coherence, extractiveness, and coverage. The study takes real-life applications into consideration by guiding the model selection for specific summarization, e.g., news summarization, legal document shortening, or medical record summarization. Another study could consider new hierarchical summarization methods and the optimization of Transformer mechanisms, which would allow to increase the efficiency and coherency of long document summarization.