Multimodal person identification in movies

Abstract

An important task for annotation of movies is finding out which characters are playing in a shot. Character identification is based on available information sources from various modalities. Fully automatic character identification is not feasible as the modalities are not semantically synchronized. As manual annotation is too time consuming, an interactive tool assisting the annotator is needed. We propose the WhoIsWho function for our interactive i-Notation system. WhoIsWho relates visual content to names extracted from movie scripts, working in both ways. We present extensive evaluation of character identification on six hours of movies. Employment of a user model enables evaluation of interactivity in WhoIsWho. Quantitative results show that WhoIsWho is successful in helping annotators identify movie characters.