Non-experts’ Trust in XAI is Unreasonably High

Abstract

The impact of explainability on users’ trust in AI has long been debated, with research often hinting that explanations of AI decisions may enhance skepticism. However, our study reveals a paradox: when faced with direct and tangible harm, non-experts continue to trust AI explanations unquestioningly. As evolving EU legislation mandates greater transparency in AI decision-making, it is critical to understand whether explainability truly enables users to detect and challenge flawed decisions. This study examines trust in explainable AI (XAI) through an experiment with 63 non-expert participants who (wrongfully) believed that an AI system was grading their exams. SHAP-like explanations accompanied the decisions, while the experimental group systematically received lower grades to simulate direct harm from simulated AI bias. Unlike prior studies relying on simulated systems, we employed a real-world high-risk use case, academic grading, where AI decisions have concrete consequences. Contrary to expectations, users’ trust levels in AI explanations remained unchanged despite clear evidence of bias, highlighting an unsettling shift from skepticism toward blind trust in XAI. These findings challenge the assumption that explainability fosters critical AI literacy and reveal a growing risk: AI explanations may reinforce misplaced trust instead of increasing caution. This underscores the urgent need to reassess how explainability is designed and whether it empowers users to engage critically with AI decisions.