Failures in the Loop: Human Leadership in AI-Based Decision-Making

Abstract

The dark side of AI has been a persistent focus in discussions of popular science and academia (Appendix A), with some claiming that AI is 'evil' [1]. Many commentators make compelling arguments for their concerns. Techno-elites have also contributed to the polarization of these discussions, with ultimatums that in this new era of industrialized AI, citizens will need to '[join] with the AI or risk being left behind' [2]. With such polarizing language, debates about AI adoption run the risk of being oversimplified. Discussion of technological trust frequently takes an all-or-nothing approach. All technologies - cognitive, social, material, or digital - introduce tradeoffs when they are adopted, and contain both 'light and dark' features [3]. But descriptions of these features can take on deceptively (or unintentionally) anthropomorphic tones, especially when stakeholders refer to the features as 'agents' [4], [5]. When used as an analogical heuristic, this can inform the design of AI, provide knowledge for AI operations, and potentially even predict its outcomes [6]. However, if AI agency is accepted at face value, we run the risk of having unrealistic expectations for the capabilities of these systems.