Error Analysis Framework for Automatic Summarization

Abstract

We propose an error analysis framework for automatic summarization. The framework presented herein incorporates five problems that cause automatic summarization systems to produce errors and three metrics for quality. We classify errors in automatic summaries into 15 categories comprising a combination of the three quality metrics and five problems. We also present a method to classify automatic-summary errors into these categories. Using our error analysis framework, we analyze the errors in an automatic summary produced by our system and present the results. We use these results to refine our system and then show that the quality of the automatic summary is improved. The error analysis framework that we propose is demonstrably useful for improving the quality of an automatic summarization system.