Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model

Abstract

Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge, which is known as the wisdom-of-the-crowd effect. However, a wide range of aggregation methods are available, which range from one-size-fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators, to customized (supervised) techniques that require past performance data, such as weighted averaging. In this study, we applied a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. We used the bias–information–noise (BIN) model to disentangle three mechanisms that allow aggregators to improve the accuracy of predictions: reducing bias and noise, and extracting valid information across forecasters. Simple averaging operates almost entirely by reducing noise, whereas more complex techniques such as prediction markets and Bayesian aggregators exploit all three pathways to allow better signal extraction as well as greater noise and bias reduction. Finally, we explored the utility of a BIN approach for the modular construction of aggregators.