A corpus of science journalism for analyzing writing quality

Abstract

We introduce a corpus of science journalism articles, categorized in three levels of writing quality. The corpus fulï¬lls a glaring need for realistic data on which applications concerned with predicting text quality can be developed and evaluated. In this article we describe how we identiï¬ed, guided by the judgements of renowned writers, samples of extraordinarily well-written pieces and how these were expanded to a larger set of typical journalistic writing. We provide details about the corpus and the text quality evaluations it can support. Our intention is to further extend the corpus with annotations of phenomena that reveal quantiï¬able differences between levels of writing quality. Here we introduce two of the many types of annotation on the sentence level that distinguish amazing from typical writing: text generality/speciï¬city and communicative goal. We explore the feasibility of acquiring annotations automatically, and verify that such features are indeed predictive of writing quality. We ï¬nd that the annotation of general/speciï¬c on sentence level can be performed reasonably accurately fully automatically, while automatic annotations of communicative goal reveals salient characteristics of journalistic writing but does not align with categories we wish to annotate in future work.