What’s Informing Trust and Adoption? Can Current Cognitive Models Support Expert Adoption of AI?

Abstract

Defence often uses rationalistic approaches for the adoption of AI, but we question whether this is appropriate. By studying adoption using naturalistic approaches in the work environment where AI is actually employed, we better can capture situated issues around trust and adoption. We conducted a longitudinal study of expert intelligence analysts adopting a machine-vision Object Detection and Recognition Overlay (ODRO) within their mission teams and tooling, premised on improving analyst functions. We employ and adapt naturalistic decision-making models to identify key situational challenges, showing that naturalistic approaches can better inform Defence’s AI adoption. Specifically, we find that organisational rewards for accuracy influence analysts’ willingness to adopt and rely on imperfect tools, and that analysts develop a non-uniform relationship with ODRO over time, which we translate into design implications. Crucially, our results support a model of interdependence over prevalent task-allocation paradigms that assume human and AI as functionally divided, rational entities.