Critical appraisal in systematic reviews of educational research—Planning steps and criteria for assessing the internal validity of intervention studies

Abstract

Systematic reviews established in evidence-based medicine are also being increasingly implemented in current educational research. Such overview studies facilitate the extensive collection of data as well as the evaluation of the effectiveness of educational measures. For this purpose, it is necessary to assess the quality of the empirical data which makes a critical appraisal of studies a central task within the review process. Unless such a review is carried out by an external systematic review organization, an internal critical appraisal of the studies should be developed independently by the researchers conducting the review. However, considering the differences in research design and in quality criteria for pedagogical intervention studies, which have to be determined in line with the evidence concept of educational research and the subject under investigation, this task poses challenges. The article takes a closer look at the development of critical appraisal in systematic reviews within educational research, especially on researchers’ assessment of the internal validity of intervention studies. We reflect on the relevant planning steps that emerge from our comprehensive search in theoretical and conceptual work as well as in tools of quality assessment. The implementation of these steps is illustrated by the example of a systematic review of content and language learning approaches. We also present the internal validity assessment tool for pedagogical intervention studies developed in this review. The article makes clear critical appraisal requires thorough planning and careful decisions concerning research direction. The set of criteria presented in this paper may serve as a guide for the quality assessment of other systematic reviews in educational research.