Automatic emotion recognition for cultural heritage

Abstract

In this work we present an automatic emotion recognition system for the re-use of multimedia content and storytelling for cultural heritage. A huge amount of heterogeneous multimedia data on cultural heritage is available in online and offline databases that can be used and adapted to produce new content. In the real world, human video editors may want to select the video sequences composing the final video with the intention to induce an emotional reaction in the viewer (e.g. happiness, excitement, sadness). Usually they try to achieve this result following their personal judgement. However, this task of video selection could benefit a lot from the exploitation of an automatic sentiment classification system. Our system can help the editor in choosing the video sequences that best fit the desired emotion to be induced. First-of-all the system splits the video in scenes. Then it classifies them using a multimodal classifier which combines temporal features extracted form LSTM, sentiment-related features obtained through a DNN, audio features and motion-related features. The system learns which features are more important and exploits them to classify the scenes in terms of valence and arousal which are well known to correlate with induced emotions. Finally it provides an online video composer which allows the editor to search, filter and compose the scenes in a new video using sentiment information. To train the classifier we also collected and annotated a small dataset of both users recorded videos and professional ones downloaded from the web.