FaraPy: An Augmented Reality Feedback System for Facial Paralysis using Action Unit Intensity Estimation

Abstract

Facial paralysis is the most common facial nerve disorder. It causes functional and aesthetic deficits that often lead to emotional distress and affect psychosocial well-being. One form of treatment is mirror therapy, which has shown potential but has several mirror-related drawbacks that limit its effectiveness. We propose FaraPy, the first mobile augmented reality mirror therapy system for facial paralysis that provides real-time feedback and tracks user progress over time. We developed an efficient convolutional neural network to detect muscle activations and intensities as users perform facial exercises in front of a mobile device camera. Our model outperforms the state-of-the-art model on benchmark data for detecting facial action unit intensity. Our user study (n=20) shows high user satisfaction and greater preference for our interactive system over traditional mirror therapy.