User Evaluation of Conversational Agents for Aerospace Domain

Abstract

The aerospace industry can benefit from conversational agents that provide efficient solutions for safety-of-life scenarios. This industry is characterized by products and systems that require years of engineering to achieve optimal performance within complex environments. With recent advances in retrieval and language models, conversational agents can be developed to enhance the system’s question-answering capabilities. However, evaluating the added-value of such systems in the context of industrial applications, such as pilots in a cockpit, can be challenging. This article presents the design, implementation, and user evaluation of a conversational agent called Smart Librarian, which is tailored to the aerospace domain’s specific requirements to support pilots in their tasks. Our results based on a controlled user experiment with flight school students indicate that the user’s perception of the usefulness of the system in completing the search task is a good predictor of both task score and time spent. The system’s responses to the relevance of the topic is also a good predictor of task score. The perceived difficulty of the search task and its interaction with the relevance of the system’s responses to the topic also play a key role in search performance. The mixed-effects models constructed in this study had large effect sizes, demonstrating participants’ ability to assess their performance accurately. Nevertheless, user satisfaction with the system’s responses may not be a reliable predictor of user search performance. Implications for the design of conversational agents based on the domain’s specific requirements are discussed.