Eat as much as you can: A Kinect-based facial rehabilitation game based on mouth and tongue movements

Abstract

In this demo, we present a Kinect-based interactive game which provides patients of facial palsy with a better and more fun way to perform facial physical therapy. By letting the user get scores when he/she bites or licks the virtual foods falling from the sky, the game goads the user into exercising their facial muscles. A robust bite action detector and an efficient tip-of-tongue locating method are developed to realize this game. Based on the head pose estimated by the depth clue, we classify the face orientation and locate the mouth region in each frame. The movements of lips are measured to detect the bite action, and different strategies are designed to more accurately detect the tip-of-tongue for various face orientations. The experimental results show that our method can be successfully applied to different users.