Assessment of writing ability in secondary education: Comparison of analytic and holistic scoring systems for use in large-scale assessments

Abstract

Although writing is an important subject of language teaching in secondary education, it is often neglected in large-scale assessments. We report results of a study with 1,365 German high school students in Grade 8 that was conducted in the context of national monitoring of educational achievement. Student responses on seven different persuasive and informative writing tasks were evaluated with two different scoring systems: (i) analytic scoring with 14 dichotomous criteria capturing specific aspects of content, text structure, and language usage, and (ii) holistic scoring based on a comprehensive rating scale similar to the NAEP Holistic Scoring Guide accom-panied by semi-holistic scales for content, style (i.e., language usage and organization), and language correct-ness. We inspected the results of both scoring procedures in terms of inter-rater and intra-rater reliability, di-mensionality, and convergence of the scoring results. Attention is also given to the generalizability of the findings across different writing tasks and text genres. The results showed better reliability for the holistic and semi-holistic scales than for most of the analytic criteria. For both scoring systems, content and structural aspects were closely associated whereas language correctness was a clearly distinct dimension. Both scoring systems measured the same latent construct.