Using StatHand to Train Structural Awareness and Promote the Development of Statistic Selection Skills

Abstract

Psychology students struggle to recall, recognize, or explain how they would select appropriate statistics for common research designs. These selection skills are underpinned by structural awareness, which is the ability to look past the surface (or topic) features of a research design and focus instead on its deep structural characteristics. Although most psychology undergraduates display limited structural awareness, it can be trained. In this preregistered experiment, we designed and evaluated a novel method of training structural awareness. This training method made use of StatHand, a free iOS and web application, in scaffolded activities designed to highlight how the structural (but not surface) characteristics of a research design determine the selection of an appropriate statistical analysis. Bayesian analyses clearly indicated that this training was effective. Specifically, trained undergraduate psychology students (n = 50) outperformed an untrained control group (n = 52) on 5 measures of structural awareness (performance on 2 sets of triad judgment tasks, 2 sets of explanation tasks, and a scenario generation task) immediately following training, and again 1 week later (δ 0.71 to 1.60). At both time points, the trained students also showed greater selection skills than the untrained control students (δ = 0.52 and 0.57). Finally, on 5 of these 6 outcome measures, the trained students showed no decrease in performance between the 2 time points. Educators are encouraged to consider how they can adapt our methods and materials for deployment in a classroom context or online activities.