Targeted school-based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K-6: A systematic review

Abstract

Background: Low levels of numeracy and literacy skills are associated with a range of negative outcomes later in life, such as reduced earnings and health. Obtaining information about effective interventions for children with or at risk of academic difficulties is therefore important. Objectives: The main objective was to assess the effectiveness of interventions targeting students with or at risk of academic difficulties in kindergarten to Grade 6. Search Methods: We searched electronic databases from 1980 to July 2018. We searched multiple international electronic databases (in total 15), seven national repositories, and performed a search of the grey literature using governmental sites, academic clearinghouses and repositories for reports and working papers, and trial registries (10 sources). We hand searched recent volumes of six journals and contacted international experts. Lastly, we used included studies and 23 previously published reviews for citation tracking. Selection Criteria: Studies had to meet the following criteria to be included:. Population: The population eligible for the review included students attending regular schools in kindergarten to Grade 6, who were having academic difficulties, or were at risk of such difficulties. Intervention: We included interventions that sought to improve academic skills, were conducted in schools during the regular school year, and were targeted (selected or indicated). Comparison: Included studies used an intervention-control group design or a comparison group design. We included randomised controlled trials (RCT); quasi-randomised controlled trials (QRCT); and quasi-experimental studies (QES). Outcomes: Included studies used standardised tests in reading or mathematics. Setting: Studies carried out in regular schools in an OECD country were included. Data Collection and Analysis: Descriptive and numerical characteristics of included studies were coded by members of the review team. A review author independently checked coding. We used an extended version of the Cochrane Risk of Bias tool to assess risk of bias. We used random-effects meta-analysis and robust-variance estimation procedures to synthesise effect sizes. We conducted separate meta-analyses for tests performed within three months of the end of interventions (short-term effects) and longer follow-up periods. For short-term effects, we performed subgroup and moderator analyses focused on instructional methods and content domains. We assessed sensitivity of the results to effect size measurement, outliers, clustered assignment of treatment, risk of bias, missing moderator information, control group progression, and publication bias. Results: We found in total 24,414 potentially relevant records, screened 4247 of them in full text, and included 607 studies that met the inclusion criteria. We included 205 studies of a wide range of intervention types in at least one meta-analysis (202 intervention-control studies and 3 comparison designs). The reasons for excluding studies from the analysis were that they had too high risk of bias (257), compared two alternative interventions (104 studies), lacked necessary information (24 studies), or used overlapping samples (17 studies). The total number of student observations in the analysed studies was 226,745. There were 93% RCTs among the 327 interventions we included in the meta-analysis of intervention-control contrasts and 86% were from the United States. The target group consisted of, on average, 45% girls, 65% minority students, and 69% low-income students. The mean Grade was 2.4. Most studies included in the meta-analysis had a moderate to high risk of bias. The overall average effect sizes (ES) for short-term and follow-up outcomes were positive and statistically significant (ES = 0.30, 95% confidence interval [CI] = [0.25, 0.34] and ES = 0.27, 95% CI = [0.17, 0.36]), respectively). The effect sizes correspond to around one third to one half of the achievement gap between fourth Grade students with high and low socioeconomic status in the United States and to a 58% chance that a randomly selected score of an intervention group student is greater than the score of a randomly selected control group student. All measures indicated substantial heterogeneity across short-term effect sizes. Follow-up outcomes pertain almost exclusively to studies examining small-group instruction by adults and effects on reading measures. The follow-up effect sizes were considerably less heterogeneous than the short-term effect sizes, although there was still statistically significant heterogeneity. Two instructional methods, peer-assisted instruction and small-group instruction by adults, had large and statistically significant average effect sizes that were robust across specifications in the subgroup analysis of short-term effects (ES around 0.35–0.45). In meta-regressions that adjusted for methods, content domains, and other study characteristics, they had significantly larger effect sizes than computer-assisted instruction, coaching of personnel, incentives, and progress monitoring. Peer-assisted instruction also had significantly larger effect sizes than medium-group instruction. Besides peer-assisted instruction and small-group instruction, no other methods were consistently significant across the analyses that tried to isolate the association between a specific method and effect sizes. However, most analyses showed statistically significant heterogeneity also within categories of instructional methods. We found little evidence that effect sizes were larger in some content domains than others. Fractions had significantly higher associations with effect sizes than all other math domains, but there were only six studies of interventions targeting fractions. We found no evidence of adverse effects in the sense that no method or domain had robustly negative associations with effect sizes. The meta-regressions revealed few other significant moderators. Interventions in higher Grades tend to have somewhat lower effect sizes, whereas there were no significant differences between QES and RCTs, general tests and tests of subdomains, and math tests and reading tests. Authors’ Conclusions: Our results indicate that interventions targeting students with or at risk of academic difficulties from kindergarten to Grade 6 have on average positive and statistically significant short-term and follow-up effects on standardised tests in reading and mathematics. Peer-assisted instruction and small-group instruction are likely to be effective components of such interventions. We believe the relatively large effect sizes together with the substantial unexplained heterogeneity imply that schools can reduce the achievement gap between students with or at risk of academic difficulties and not-at-risk students by implementing targeted interventions, and that more research into the design of effective interventions is needed.