Validating a Holistic Rubric for Scoring Short Answer Reading Questions

Abstract

This study presents an investigation of the validity of a practical scoring procedure for short answer (sentence length) reading comprehension questions, using a holistic rubric for sets of responses rather than scoring individual responses, on a university-based English proficiency test. Thirty-three previously scored response sets were rescored by the researchers at the item level, with ratings for completion, fidelity to the source text, and overlap with source text language. Results confirm that holistic section-based scores are predicted by fidelity ratings only, providing evidence to support the scoring inference of the validity argument.