PreSS: Predicting Student Success Early in CS1. A Pilot International Replication and Generalization Study

Abstract

This work piloted an international replication and generalization study on an existing prediction model called PreSS. PreSS has been developed and validated over nearly two decades and can predict student performance in CS1 with nearly 71% accuracy, at a very early stage in the learning module. Motivated by a prior validation study and its competitive modelling accuracy, we chose PreSS for such an international replication and generalization study. The study took place in two countries, with two institutions in Ireland and one institution in the US, totalling 472 students throughout the academic year 2020-21. In doing so, this study addressed a call from the 2015 ITiCSE working group for the educational data mining and learning analytics community: systematically analyse and verify previous studies using data from multiple contexts to tease out tacit factors that contribute to previously observed outcomes. This pilot study achieved 90% accuracy, which is higher than the prior work's. This encouraging finding sets the foundations for a larger scale international study. This paper describes in detail the pilot replication and generalization study and our progress on the larger scale study which is taking place across six continents.