A crisis of overconfidence: Why confidence, not accuracy, is the real risk in clinical AI

Abstract

Language models today are trained to convey confidence in their outputs, regardless of whether those outputs are correct. The alignment methods we use to make them helpful also push them toward unwarranted certainty, rewarding decisive answers over appropriate hedging. As these foundation models enter high-stakes domains such as science and medicine, this disconnect between how sure they sound and how accurate they are can become dangerous. Here, we examine why post-training degrades a model’s sense of uncertainty, and we review techniques that can bring expressed confidence back in line with actual reliability. Through this, we argue that trustworthy AI means treating calibration as a core design goal.