Replication unreliability in psychology: Elusive phenomena or "elusive" statistical power?

Abstract

The focus of this paper is to analyze whether the unreliability of results related to cer- tain controversial psychological phenomena may be a consequence of their low statistical power. Applying the Null Hypothesis StatisticalTesting (NHST), still the widest used statisti- cal approach, unreliability derives from the failure to refute the null hypothesis, in particular when exact or quasi-exact replications of experiments are carried out. Taking as example the results of meta-analyses related to four different controversial phenomena, sublimi- nal semantic priming, incubation effect for problem solving, unconscious thought theory, and non-local perception, it was found that, except for semantic priming on categorization, the statistical power to detect the expected effect size (ES) of the typical study, is low or very low. The low power in most studies undermines the use of NHST to study phenom- ena with moderate or low ESs. We conclude by providing some suggestions on how to increase the statistical power or use different statistical approaches to help discriminate whether the results obtained may or may not be used to support or to refute the reality of a phenomenon with small ES. © 2012 Tressoldi.