Semantic correlation mining between images and texts with global semantics and local mapping

Abstract

This paper proposes a novel approach for the modeling of semantic correlation between web images and texts. Our approach contains two processes of semantic correlation computing. One is to find the local media objects (LMOs), the components composing text (or image) documents, that match the global semantics of a given image(or text) document based on probabilistic latent semantic analysis (PLSA); The other is to make a direct mapping among LMOs with graph-based learning, with those LMOs achieved based on PLSA as a part of inputs. The two cooperating processes consider both dominant semantics and local subordinate parts of heterogeneous data. Finally, we compute the similarity between the obtained LMOs and a whole document of the same modality and then get the semantic correlation between textual and visual documents.Experimental results demonstrate the effectiveness of the proposed approach.