Movie Content Analysis, Indexing and Skimming Via Multimodal Information

Abstract

A content-based movie analysis, indexing and skimming system is developed in this research. Specifically, it includes the following three major modules: 1) an event detection module, where three types of movie events, namely, two-speaker dialogs, multiple-speaker dialogs, and hybrid events are extracted from the content. Multiple media cues such as audio, speech, visual and face information are integrated to achieve this goal; 2) a speaker identification module, where an adaptive speaker identification scheme is proposed to recognize target movie cast members for content indexing purposes. Both audio and visual sources are exploited in the identification process, where the audio source is analyzed to recognize speakers using a likelihood-based approach, and the visual source is examined to locate talking faces with face detection/recognition and mouth tracking techniques; 3) a movie skimming module, where an event-based skimming system is developed to abstract movie content in the form of a short video clip for content browsing purposes. Extensive experiments on integrating multiple media cues for movie content analysis, indexing and skimming have yielded encouraging results.