Considerations in the interpretation of cosmological anomalies

Abstract

Anomalies drive scientific discovery - they are associated with the cutting edge of the research frontier, and thus typically exploit data in the low signal-to-noise regime. In astronomy, the prevalence of systematics - both known unknowns and unknown unknowns - combined with increasingly large datasets, the widespread use of ad hoc estimators for anomaly detection, and the look-elsewhere effect, can lead to spurious false detections. In this informal note, I argue that anomaly detection leading to discoveries of new physics requires a combination of physical understanding, careful experimental design to avoid confirmation bias, and self-consistent statistical methods. These points are illustrated with several concrete examples from cosmology.