Interpreting the effectiveness of a summer reading program: The eye of the beholder

Abstract

In applying a methods-oriented approach to evaluation, this study interpreted the effectiveness of a summer reading program from three different stakeholder perspectives: practitioners from the school district, the funding agency supporting the program, and the policymakers considering mandating summer school. Archival data were obtained on 2330 students reading below benchmark in Grades 2–5. After propensity score matching participants to peers who did not attend the summer program, the final sample consisted of 630 students. Pre-to-posttest growth models revealed positive effects in Grades 2–4 (standardized slopes of .40–.54), but fifth graders demonstrated negligible improvement (standardized slope of .15). The standardized mean differences of propensity score matched treatment and control group students indicated null effects in all grade levels (d = −.13 to .05). Achieving proficient reading performance also was not attributable to summer school participation. Findings underscore the importance of operationalizing effectiveness in summative evaluation.