Real time multimodal emotion recognition system using facial landmarks and hand over face gestures

Abstract

Over the last few years, emotional intelligent systems have changed the way humans interact with machines. The main intention of these systems is not only to interpret human affective states but also to respond in real time during assistive human to device interactions. In this paper we propose a method for building a Multimodal Emotion Recognition System (MERS), which combine mainly face cues and hand over face gestures which work in near real time with an average frame rate of 14 Fps. Although there are many state of the art emotion recognition systems using facial landmarks, we claim that our proposed system is one of the very few which also include hand over face gestures, which are commonly expressed during emotional interactions.