Composite AI for Behavior Analysis in Social Interactions

Abstract

Social interactions are fundamental to human life. Accurately identifying and interpreting verbal and non-verbal cues is essential for analyzing human behavior and human-machine interactions. The complexity of these interactions, along with the different communication signals, and their varying frequencies is a challenge that Deep Neural Networks cannot yet address. Composite AI, which combines Deep Learning (DL) and traditional AI methods, emerges as a potential approach. We propose a framework comprising three main modules: feature extraction, episode detection, and activity interpretation. The feature extraction module uses DL methods to extract relevant information about the environment, interacting partners, and context of the interaction. By combining these features with rule-based systems, we restrict the analysis to key episodic components, effectively reducing the false positive detection rate. Classifiers are then used to identify and recognize specific sub-events. We show our system's effectiveness in detecting communication cues on recorded sessions of a standardized autism diagnostic test (ADOS-2) involving autistic children, where detecting such cues can be highly challenging. In this scenario, we achieve 87% and 89% accuracy for verbal and non-verbal requests, respectively, and zero false positives. This comprehensive framework can significantly enhance social interaction analysis, enabling more effective automated event analysis. We also highlight the importance of sub-event segmentation and the components that could be improved with additional algorithmic combinations or data-driven improvements of the detection systems.