A Study on the Development of a Mixed Reality System Applied to the Practice of Socially Interactive Behaviors of Children with Autism Spectrum Disorder

Abstract

Before entering the adult world, typically developing (TD) children tend to learn how to socialize with others in the process of getting along and interacting with their peers at school. However, children with autism spectrum disorder (ASD) have difficulty understanding others’ emotions and discerning non-verbal social clues due to their innately impaired social interaction and lack of theory of mind (ToM). Therefore, while socializing with others, children with ASD might not understand the facial expressions or the body movements of others. This causes great difficulty in discerning other people’s emotions and recognizing the relationships between themselves and others when interacting with different people in different situations, and as a consequence, to perform the appropriate social reciprocal acts. Therapists or special education teachers usually conduct role-playing strategies to train children with ASD to practice their social reciprocity behavior, after which these children with ASD can practice socializing with others. In reality, however, such a method is generally limited to a single classroom environment and specific situations. For unimaginative children with ASD, such intervention situations and teaching methods are typically limited with low effectiveness. In response, this study adopted mixed-reality (MR) technology to establish a semi-immersive social-interactive situated teaching platform for training emotional representations and non-verbal social cues. The system allows children with ASD to socialize with 3D virtual characters who have different relationships in different virtual situations, enabling them to practice performing the appropriate social reciprocal acts.