How much evidence is p >.05? Stimulus pre-testing and null primary outcomes in violent video games research.

Abstract

Research on the effects of violent video games frequently relies on arguments for the null hypothesis. Proponents of the effects argue that there are no meaningful differences save violent content between the violent and nonviolent games played, while critics of the effects argue that their nonsignificant study results constitute evidence for the null hypothesis of no difference. However, neither argument can be supported through the use of traditional null-hypothesis significance testing, as such tests can only ever reject or retain the null, never rejecting the alternative hypothesis in favor of the null. Therefore, to evaluate these claims, we apply a more appropriate Bayesian analysis to measure evidence for or against the null hypothesis relative to reasonable alternative hypotheses. We conclude that current methodological standards cannot rule out substantial confounds between violent and nonviolent video games. Furthermore, we find that studies that claim to find an absence of violent video game effects vary substantially in the strength of evidence, with some strongly supporting the null, others weakly supporting the null, and some others finding evidence of differences between conditions. We recommend the use of Bayesian analyses, larger sample sizes, and the creation of custom-designed games for experimental research.