Interrater reliability in second language meta-analyses

Abstract

There has recently been a surge of interest in improving the replicability of second language (L2) research. However, less attention is paid to replicability in the context of L2 meta-analyses. I argue that conducting interrater reliability (IRR) analyses is a key step toward improving the replicability of L2 meta-analyses. To that end, I first discuss the foundation of IRR in the context of meta-analytic research. Second, I introduce two IRR measures, S index and Specific Agreement, which aid in improving the replicability of meta-analytic research. Third, I offer a flexible R program, meta_rate, to facilitate the conduct of IRR analyses for L2 meta-analyses. Fourth, I apply the R program to an actual meta-analytic L2 coding sheet to demonstrate the practical use of the IRR methods discussed. Finally, I provide interpretive guidelines to assist both L2 meta-analysts and journals with the transparent reporting of the IRR findings.