Towards an Augmented Reality Agent to Support Communication for Nonspeaking Autistic People

Abstract

Many nonspeaking autistic people have learned to communicate using a letterboard. With this technique, a trained Communication and Regulation Partner (CRP) holds a plastic letterboard and reads aloud the letters an individual is pointing toward. Training to communicate in this manner is typically a lengthy process and the cost and availability of a professional CRP can be a limiting factor to an individual's ability to progress. To address this problem, we build a system that allows an individual to wear an Augmented Reality (AR) headset and engage in spelling exercises on a holographic letterboard supported by a fully embodied, virtual CRP. Specifically, we video recorded 20-minute spelling sessions collected from 13 participants and coded these videos to understand the nature of interactions between a user and their CRP. Later, we use the coded data to derive a Markov decision process behaviour model that allows the virtual CRP (ViC) to mimic a real CRP. Our preliminary results show that ViC can achieve an accuracy of up to 68% even with the limited data we were able to collect and the simplified modeling approach we used. Thus, our work opens doors for nonspeakers to practice various aspects of spelling-based communication in the absence of a human CRP, thereby fostering increased independence and privacy.