"i'd like an Explanation for That!" Exploring reactions to unexpected autonomous driving

Abstract

Autonomous vehicles are complex systems that may behave in unexpected ways. From the drivers' perspective, this can cause stress and lower trust and acceptance of autonomous driving. Prior work has shown that explanation of system behavior can mitigate these negative effects. Nevertheless, it remains unclear in which situations drivers actually need an explanation and what kind of interaction is relevant to them. Using thematic analysis of real-world experience reports, we first identified 17 situations in which a vehicle behaved unexpectedly. We then conducted a think-aloud study (N = 26) in a driving simulator to validate these situations and enrich them with qualitative insights about drivers' need for explanation. We identified six categories to describe the main concerns and topics during unexpected driving behavior (emotion and evaluation, interpretation and reason, vehicle capability, interaction, future driving prediction and explanation request times). Based on these categories, we suggest design implications for autonomous vehicles, in particular related to collaboration insights, user mental models and explanation requests.