Visual video analytics for interactive video content analysis

Abstract

Reasoning as an essential processing step for any data analysis task, yet it requires semantic, contextual understanding on a high level, e.g., for the identification of entities. Developing an architecture for visual video analytics (VVA), we integrate human knowledge for highly accurate video content analysis to extract information by a tight coupling of automatic video analysis algorithms on the one hand and visualization as well as user interaction on the other hand. For accurate video content analysis, our semi-automatic VVA-architecture effectively understands and identifies regular and irregular behavior in real-world datasets. The VVA-architecture is described with both (i) its interactive information extraction and representation and (ii) its content-based reasoning process. We give an overview of existing techniques for information extraction and representation, and propose two interactive applications for reasoning. One of the applications uses 3D object representations to provide adaptive playback based on selected object parts in the 3D viewer. Another application allows the formulation of a proposition about the video by using all extracted objects and information. In case the proposition is correct, the corresponding frames of the video are highlighted. Based on a user study, relevant open topics for increasing the performance of video content analysis and VVA is discussed.