Gesture and Voice Commands to Interact With AR Windshield Display in Automated Vehicle: A Remote Elicitation Study

Abstract

Augmented reality (AR) windshield display (WSD) offers promising ways to engage in non-driving tasks in automated vehicles. Previous studies explored different ways WSD can be used to present driving and other tasks-related information and how that can affect driving performance, user experience, and performance in secondary tasks. Our goal for this study was to examine how drivers expect to use gesture and voice commands for interacting with WSD for performing complex, multi-step personal and work-related tasks in an automated vehicle. In this remote unmoderated online elicitation study, 31 participants proposed 373 gestures and 373 voice commands for performing 24 tasks. We analyzed the elicited interactions, their preferred modality of interaction, and the reasons behind this preference. Lastly, we discuss our results and their implications for designing AR WSD in automated vehicles.