Enhanced sports image annotation and retrieval based upon semantic analysis of multimodal cues

Abstract

This paper presents a framework for semi-automatic annotation and semantic image retrieval, applied to the sports domain, based upon semantic analysis of both image text captions and visual features of the image. Unstructured text captions of images are analysed in order to extract the concepts and restructure them into a semantic model. SVM classification of the multi-dominant colours and edge ratio information of the images are used to classify the sport genre. The novelty of the proposed semantic framework is that it can find both the indirectly relevant concepts (concepts not directly referred to) in the visual information and can represent the semantic of images at a higher level by combining image captions and visual feature information. In addition, integrating LSI into the semantic framework enables the proposed system to tolerate ontology imperfections. Experimental results show that the use of the semantic approach significantly enhances image retrieval. Semantic visual information classification and retrieval based upon multimodal cues. © 2009 Springer Berlin Heidelberg.