How Do Computing Education Researchers Talk About Threats and Limitations?

Abstract

Background and Context: Empirical researchers have a long-standing tradition of explicitly discussing the threats to and limitations of their research. In the past twenty years, these discussions emerged as a standard component of empirical research papers in computing education research (CER) as well. Objective: Our goal was to find out how the CER community talks about threats and limitations (which we refer to collectively as "challenges") and how they "respond"to these challenges. Method: Our dataset included a total of 77 papers from four venues: two CER journals, Computer Science Education (CSEJ) and ACM Transactions of Computing Education (TOCE); one CER conference, ACM Conference on International Computing Education Research (ICER); and, for comparison, one mathematics-education journal, The Journal for Research in Mathematics Education (JRME). We analyzed the discussions of threats and limitations in these papers using deductive codes drawn from the literature, while also being open to new codes that emerged from the data. We took the papers on their own terms, so where qualitative and quantitative papers discuss challenges differently, we report on both. Findings: We found that the majority of these papers - 65 out of 77 - did discuss challenges. Depending on the research methodology employed, authors reported challenges that we could map to either internal and external validity, construct validity, and statistical conclusions or to trustworthiness criteria. Most of the challenges related to study design, but some related to study implementation, analysis, and interpretation of the results. Almost none of the challenges we found were unique to CER, but were also found in JRME. Our contributions include describing the broad range of challenges and responses that are discussed and connecting them to concrete circumstances in CER. Finally, we find that threats are seen both as important information about and as challenges to the study being reported. The tension between these two perspectives is resolved by including both, in patterns of challenge and response. Implications: Most immediately, we hope that this paper will broaden the perspective of those sitting down to write a threats and limitations section. It may also be of use at the study-design phase, since some of the challenges can be avoided with the benefit of hindsight. And finally, we hope to start a conversation about the challenges to our research: which can be mitigated, when (and how) a convincing argument can be made that an apparent challenge is not a problem, and which challenges might warrant serious revisions before submission (by the author) or rejection (by reviewers).