Real-time simulation of virtual humans' emotional facial expressions, harnessing autonomic physiological and musculoskeletal control

Abstract

Emotion expressions and communication involve involuntary and voluntary processes that may not always operate consistently. Expressive behaviour involving physiological changes may carry information about hidden aspects of emotional experience important for accurate emotion recognition and social cognition. In this paper, we describe a principled approach to simulating Virtual Human physiological facial features related to the autonomic nervous system (ANS). Our approach is based on typical synergies within the two branches of the ANS. It covers both parasympathetic tone and sympathetic tone, and their impact on skin tone, pupil diameter and sweat. We also present the triggering of tears. We discuss the implementation of the approach as part of a 3D toolkit. This work is aimed at supporting the development of affective features for realtime intelligent artificial agents, and the study of the perception of mixed emotion and emotion regulation. We demonstrate how varying ANS parameters impacts facial behaviour, contrasting emotionally consistent vs inconsistent musculoskeletal and ANS-related features.