Comparing evidence on the effectiveness of reading resources from expert ratings, practitioner judgements, and research repositories

Abstract

Background: Education decision makers are increasingly expected to use evidence to inform their actions. However, the majority of educational interventions have not yet been studied and it is challenging to produce high quality research evidence quickly enough to influence policy questions. Aims and objectives: We set out to gather evidence on the efficacy of reading resources implemented at 23 struggling elementary schools in a large, urban district in the US. The schools were at risk of closure by the state. Methods: For each reading resource, we searched for existing effectiveness studies and collected professional judgements by surveying practitioners. We also used an expert survey to collect judgements from three reading experts. We compared the ratings among experts and between practitioners and experts. We also compared practitioner and expert judgements to evidence summaries from research repositories. Findings: We found evidence summaries in research repositories for only five of 23 reading resources used in the 23 schools. Experts showed poor to good agreement on ten questions about each resource. Agreement between practitioners and experts was low with practitioners generally rating resources more positively than reading experts. Discussion and conclusions: Practitioners may be overly optimistic about the efficacy of educational materials while experts have difficulty assessing how well the materials serve an unfamiliar population. In the absence of rigorous evaluations of locally-implemented programmes, district decision makers can review the consistency of evidence collected from practitioners and experts, along with external research evidence, to inform actions towards supporting and guiding struggling schools.