Auto-annotation of paintings using social annotations, domain ontology and transductive inference

Abstract

Knowledge of paintings domain includes a variety of sources such as essays, visual examples, ontologies of artistic concepts and user- provided annotations. This knowledge serves several purposes. First, it defines a wide range of concepts for annotation and flexible retrieval of paintings. Second, it serves to bootstrap auto-annotation and disambiguate the generated candidate labels. Third, the user-provided annotations serve to discover folksonomies of concepts and vernacular terms. In this paper, we propose a framework for paintings auto-annotation that incorporates user provided images and annotations, domain ontology and external knowledge sources. We utilize these sources of information to bootstrap and support the auto-annotation task, which is based on transductive inference mechanism that combines probabilistic clustering and multi-expert approach to generate labels. We further combine user-provided annotations with generated labels and domain ontology to disambiguate the concepts. In our experiments, we focus on the auto-annotation of painting and demonstrate that the user-provided annotations significantly increase annotation accuracy. © Springer-Verlag Berlin Heidelberg 2007 Transductive Inference, Multi-expert Approach, Probabilistic Clustering,.