Socially Informed AI for Healthcare: Understanding and Generating Multimodal Nonverbal Cues

Abstract

Advances in the areas of face and gesture analysis, computational paralinguistics, multimodal interaction, and human-computer interaction have all played a major role in shaping research into assistive technologies over the last decade. This has resulted in a breadth of practical applications ranging from diagnosis and treatment tools to social companion technologies. From an analytical perspective, nonverbal cues provide understanding into the assessment of mental health and wellbeing (i.e., detecting depression and pain) and the detection of developmental and neurological conditions such as autism, dementia, and schizophrenia. From both a synthesis and generative perspective, it is necessary that assistive technologies, either disembodied or embodied, are capable of generating engaging, interactive behaviours and interventions that are personalised and adapted to user's needs, profiles, and preferences. While nonverbal cues play an essential role, there are still many key issues to overcome, which affect both the development and the deployment of multimodal technologies in real-world settings. The key aim of this multidisciplinary workshop is to foster cross-pollination by bringing together computer scientists and social psychologists to discuss innovative ideas, challenges and opportunities for understanding and generating multimodal nonverbal cues within the scope of healthcare applications1.