The Wisdom of the Coherent: Improving Correspondence With Coherence-Weighted Aggregation

Abstract

Previous research shows that variation in coherence (i.e., degrees of respect for axioms of probability calculus), when used as a basis for performance-weighted aggregation, can improve the accuracy of probability judgments. However, many aspects of coherence-weighted aggregation remain a mystery, including both prescriptive issues (e.g., how best to use coherence measures) and theoretical issues (e.g., why coherenceweighted aggregation is effective). Using data from six experiments in two earlier studies (N = 58, N = 2,858) employing either general-knowledge or statistical information integration tasks, we addressed many of these issues. Of prescriptive relevance, we examined the effectiveness of coherence-weighted aggregation as a function of judgment elicitation method, group size, weighting function, and the bias of the function’s tuning parameter. Of descriptive relevance, we propose that coherenceweighted aggregation can improve accuracy via two distinct, task-dependent routes: a causal route in which the bases for scoring accuracy depend on conformity to coherence principles (e.g., Bayesian information integration) and a diagnostic route in which coherence serves as a cue to correct knowledge. The findings provide support for the efficacy of both routes, but they also highlight why coherence weighting, especially the most biased forms, sometimes imposes costs to accuracy. We conclude by sketching a decision–theoretic approach to how aggregators can sensibly leverage the wisdom of the coherent within the crowd.