Automatic generation of meta-summaries for evaluation of the handling of discursive structures and coherence in students

Abstract

Crowd-sourcing can help teachers to evaluate student summaries and give them feedback to improve their summarization skills. In this paper, we propose an approach for meta-summaries generation, to design and develop the automatic evaluation of extractive summaries for the Basque language. We propose a novel algorithm that allows to use the generated meta-summaries to i) compare students meta-summaries at different ages and education stages (elementary and undergraduates), ii) evaluate classroom meta-summaries (classroom evaluation) and iii) evaluate each student (individual evaluation). The results show that our proposed method, based on qualitative (coherence discourse structure) and quantitative (Fleis kappa and Hamming distance) measures, is accurate to compare both: groups and individuals.