A structured learning approach to semantic photo indexing and query

Abstract

Consumer photos exhibit highly varied contents, diverse resolutions and inconsistent quality. The objects are usually ill-posed, occluded, and cluttered with poor lighting, focus, and exposure. Existing image retrieval approaches face many obstacles such as robust object segmentation, small sampling problem during relevance feedback, semantic gap between low-level features and high-level semantics, etc. We propose a structured learning approach to design domain-relevant visual semantics, known as semantic support regions, to support semantic indexing and visual query for consumer photos. Semantic support regions are segmentation-free image regions that exhibit semantic meanings and that can be learned statistically to span a new indexing space. They are detected from image content, reconciled across multiple resolutions, and aggregated spatially to form local semantic histograms. Query by Spatial Icons (QBSI) is a unique visual query language to specify semantic icons and spatial extents in a Boolean expression. Based on 2400 heterogeneous consumer photos and 26 semantic support regions learned from a small training set, we demonstrate the usefulness of the visual query language with 15 QBSI queries that have attained high precision values at top retrieved images. © Springer-Verlag Berlin Heidelberg 2005.