Publication Policies for Replicable Research and the Community-Wide False Discovery Rate

Abstract

Recent literature has shown that statistically significant results are often not replicated because the “p-value < 0.05” publication rule results in a high false positive rate (FPR) or false discovery rate (FDR) in some scientific communities. While recommendations to address the phenomenon vary, many amount to incorporating additional study summary information, such as prior null hypothesis odds and/or effect sizes, in some way. This article demonstrates that a statistic called the local false discovery rate (lfdr), which incorporates this information, is a sufficient summary for addressing false positive rates. Specifically, it is shown that lfdr-values among published results are sufficient for estimating the community-wide FDR for any well-defined publication policy, and that lfdr-values are sufficient for defining policies for community-wide FDR control. It is also demonstrated that, though p-values can be useful for computing an lfdr, they alone are not sufficient for addressing the community-wide FDR. Data from the recent replication study are used to compare publication policies and illustrate the FDR estimator.