Simulation-based inference has its own Dodelson–Schneider effect (but it knows that it does)

Abstract

Context. Making inferences about physical properties of the Universe requires knowledge of the data likelihood. A Gaussian distribution is commonly assumed for the uncertainties with a covariance matrix estimated from a set of simulations. The noise in such covariance estimates causes two problems: it distorts the width of the parameter contours, and it adds scatter to the location of those contours that is not captured by the widths themselves. For non-Gaussian likelihoods, an approximation may be derived via simulation-based inference (SBI). It is often implicitly assumed that parameter constraints from SBI analyses, which do not use covariance matrices, are not affected by the same problems as parameter estimation with a covariance matrix estimated from simulations. Aims. We aim to measure the coverage and marginal variances of the posteriors derived using density-estimation SBI over many identical experiments to investigate whether SBI suffers from effects similar to those of covariance estimation in Gaussian likelihoods. Methods. We used a neural-posterior and likelihood estimation with continuous and masked autoregressive normalising flows for density estimation. We fitted our approximate posterior models to simulations drawn from a Gaussian linear model, so the SBI result can be compared to the true posterior, and effects related to noise in the covariance estimate are known analytically. We tested linear and neural-network-based compression, demonstrating that neither method circumvents the issues of covariance estimation. Results. SBI suffers an inflation of posterior variance that is equal to or greater than the analytical result in covariance estimation for Gaussian likelihoods for the same number of simulations. This inflation of variance is captured conservatively by the reported confidence intervals, leading to an acceptable coverage regardless of the number of simulations. The assumption that SBI requires a smaller number of simulations than covariance estimation for a Gaussian likelihood analysis is inaccurate. The limitations of traditional likelihood analysis with simulation-based covariance remain for SBI with finite simulation budget. Despite these issues, we show that SBI correctly draws the true posterior contour when there are enough simulations.