Diagnosing Data Analytic Problems in the Classroom

Abstract

Teaching data analysis by providing students with real-world problems and datasets allows students to integrate a variety of skills in a situation that mirrors how data analysis actually works. However, whole data analyses may obscure the individual skills of data analytic practice that are generalizable across data analyses. One such skill is the ability to diagnose the cause of unexpected results in a data analysis. While experienced analysts can quickly iterate through a series of potential explanations when confronted with unexpected results, novice analysts often struggle to figure out how to move forward. The goal of this article is to describe an approach to teaching students skills in diagnosing data analytic problems. The exercise described here is targeted to allow students to practice this skill and to assess the depth of their knowledge about the statistical tools they have learned. We take a hypothetical case study approach and focus on the students’ reasoning through their diagnoses and suggestions for follow-up action. We found the implementation of this exercise in a small graduate course to provide valuable information about the students’ diagnostic thought processes, but further work is needed regarding structured approaches to implementation and the design of assessments. Supplementary materials for this article are available online.