Metascientific replication project with the advanced meta-experimental protocol of the transparent psi project procedures for testing the precognitive effect claimed by Bem

Abstract

This metascientific project studied the replicability of Bem Experiment 1, which had claimed a precognitive effect, i.e., the ability to successfully guess the outcome of future random events (Bem. J Pers Soc Psychol. 2011;100: 407−25). The use of advanced methodologies—based on the advanced meta-experimental protocol (AMP) and transparent psi project (TPP) procedures—reduced the risk of false discoveries as a function of (i) confirmation bias, (ii) non-transparency, and (iii) intrinsic measurement bias. The combined AMP-TPP test strategy performed three replication studies with a total of 26,483 participants resulting in N=420,472 critical trials. Study 1 failed to replicate the precognitive effect. An exploratory analysis of Study 1 suggested an effect in the opposite direction than was originally predicted (49.48% ± 0.26 SE; N = 37,836). Study 2 confirmed this exploratory result using a high-powered replication design (49.65% ± 0.14 SE; p = 0.013; N = 127,000). Study 3 was unable to replicate the result from Study 2 (50.07% ± 0.11 SE; p = 0.496; N = 217,800). The results of Study 2 represent a rare example in psi research of the successful replication of an exploratory result using a confirmatory protocol. The source of the one-time confirmed anomalous result in Study 2 remains to be identified. This result presents either (i) a psi-derived anomaly that defies known physical laws, or (ii) a method-derived anomaly, e.g., a false-positive statistical finding. Using conventional standards, based on the lack of replicability in Study 3 and absence of an accepted scientific theory, the second scenario appears more plausible. This AMP-TPP metascientific project demonstrated the use of advanced controls for assessing the reliability of the employed scientific process. This project shows how rigor-enhancing test strategies can improve the reliability, not only of psi research, but any type of weak-effects experiments, including in psychology.