The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)

Abstract

An article by Kim et al. from 2014 examined individual- and school-level variables affecting the information and communication technology (ICT) literacy level of Korean elementary school students, finding differential gender effects. In this secondary data replication, we used data from the 2018 International Computer and Information Literacy Study, focusing on data from Korea as main replication. As many characteristics of the study as possible, such as variables and analytical strategy, were modelled in the analysis. Additional analyses included 13 countries and jurisdictions, varied centring techniques for variables, and missing data treatment. The replication and analyses were pre-registered via the Open Science Framework. The main analysis did not replicate the main gender finding. However, it was also clear that, despite care taken in a rigorous replication, analytical variability still plays a large role in replications of findings, and with secondary datasets. We discuss the implications of this for secondary data replications.