Automated Behavior Labeling During Team-Based Activities Involving Neurodiverse and Neurotypical Partners Using Multimodal Data

Abstract

The employment setting for autistic individuals in the USA is grim. Based on reports, individuals with ASD struggle to secure and retain employment due to challenges in communicating and collaborating with others in workplace settings which is often attributed to their social skills deficit. Current programs that support collaborative skills development in vocational settings rely on manual evaluation and feedback by human observers, which can be resource straining and receptive to bias. Using a collaborative virtual environment (CVE) allows neurodiverse individuals to develop teamwork skills by working together with a neurotypical partner in a shared virtual space. An effective CVE system can provide real-time prompts by recognizing the user’s behavior to promote teamwork. As such, it is crucial to be able to automatically label both users’ behaviors. In this paper, we propose using K-means clustering to automate behavior labeling in a workplace CVE. The results show that K-means clustering enables high accuracy in predicting the user’s behavior, therefore, confirming that it can be used in future studies to support real-time prompts to encourage teamwork in a CVE.