Video summarization for expression analysis of motor vehicle operators

Abstract

We develop a mobile face analysis system to detect the stress of motor vehicle operators. This system has the potential to predict and notify the driver when their stress has reached a level that may affect their ability to drive. The primary goal is software that has reduced computational requirements to be deployed in a mobile environment. For a single subject, not all frames are needed to characterize the emotion in the scene. Some expressions may be spurious, neutral, or repetitive and reduce prediction accuracy. To this end, we investigate the importance of video summarization for facial emotion recognition in mobile applications. We detail a novel algorithm that succinctly describes an entire frontal face video. Previous work determines the minimal sampling rate needed for facial expressions, but summarization occurs at evenly spaced intervals that might not align with frames where expressions are the most visible. Minimum Sparse Representation selects exemplar frames where expression is most prominent. However, the sampling rate is not based on the frequency of expressions. We propose a novel algorithm that combines both approaches: an appropriate sampling rate is determined for each video clip and frame exemplars are selected at dynamic intervals. The proposed method improves accuracy over four other video summarization algorithms on a real-world data set from Motor Trend Magazine’s Best Driver Car. The approach reduces the number of frames required by 83.21% from 308,202 to 51,739, while reducing mean squared error by 61.87%.