Confidence-Based Trust Calibration in Human-AI Teams

Abstract

Effective human-AI collaboration is contingent upon calibrated trust, wherein users depend on AI systems when accuracy is probable and rely on human judgment when errors are likely. In this study, a confidence-based mechanism for trust calibration within human-AI teams is examined. A decisionmaking strategy is proposed in which task delegation is governed by the AI’s confidence: when the confidence surpasses a specified threshold, the AI’s recommendation is adopted, otherwise, the decision is deferred to the human. Through simulation experiments on a binary classification task, performance outcomes are compared. The AI system achieves an accuracy of 77.7%, whereas the human decision-maker, modeled with a confidencesensitive accuracy function ph(c) = 0.95 − 0.3c, attains an overall accuracy of 71.9%. Team performance is evaluated across a range of AI confidence thresholds (0.50 to 0.99), revealing that an intermediate threshold yields optimal team accuracy of 84.14%, substantially exceeding the performance of either agent individually. The findings provide a detailed analysis of confidence-based delegation, align with existing research on trust calibration, and underscore critical design implications for the development of human-centric AI systems.