Data-driven unsupervised evaluation of automatic text summarization systems

Abstract

Automatic text summarization is a text compression problem with many applications in natural language processing. In this paper we focus the problem of the evaluation of text summarization system. We propose an unsupervised approach based on keywords: it does not require large amount of manual processing and can be implemented as a fully automatic procedure. We also conduct a series of experiments with naïve informants and professional experts. The results of the experiments with informants, experts and automatically extracted keywords confirm that keywords, as one of the types of text compression, can be successfully used for the evaluation of summaries quality. Our data is represented by (but not restricted to) different types of Russian news texts.