Is There a Reproducibility Crisis? On the Need for Evidence-based Approaches

Abstract

The ‘Sixth Report—Reproducibility and Research Integrity’ (UK House of Commons Science, Innovation and Technology Committee 2023. ‘Sixth Report—Reproducibility and Research Integrity’) (‘The Report’) recommends measures designed to tackle an alleged ‘reproducibility crisis’ in scientific research. Our systematic analysis of the content of this report revealed that its findings and recommendations are consistent with the scientific literature, including the acknowledgement that conclusive evidence demonstrating the existence of a ‘reproducibility crisis’ is lacking. Though conceding that there is currently no way to determine the size of the crisis or whether it even exists, The Report nevertheless proposes actions to tackle the alleged crisis. However, without a quantitative understanding, the efficacy of the proposed measures cannot be verified. Hence, the current approach towards the alleged reproducibility crisis, here exemplified by The Report, does not adhere to the standards that would normally applied to the scientific method. An evidence-based approach requires the establishment of a quantitative understanding of whether data variability in the research literature exceeds technically achievable levels of reproducibility. If it does, the resulting understanding will enable the design of actions, whose success can be monitored. Our findings emphasise that the research environment requires the same level of rigour and scrutiny as the scientific experiments themselves.