Picture it in your mind: generating high level visual representations from textual descriptions

Carrara F, Esuli A, Fagni T, Falchi F, Moreo Fernández A ...see all

Information Retrieval Journal (2017) pp. 1-22 Published by Springer Netherlands

  • 26

    Readers

    Mendeley users who have this article in their library.
  • 0

    Citations

    Citations of this article.
Sign in to save reference

Abstract

In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the, typically huge, image collection on which the search is performed. We propose Text2Vis, a neural network that generates a visual representation, in the visual feature space of the fc6-fc7 layers of ImageNet, from a short descriptive text. Text2Vis optimizes two loss functions, using a stochastic loss-selection method. A visual-focused loss is aimed at learning the actual text-to-visual feature mapping, while a text-focused loss is aimed at modeling the higher-level semantic concepts expressed in language and countering the overfit on non-relevant visual components of the visual loss. We report preliminary results on the MS-COCO dataset.

Author-supplied keywords

  • Cross-media retrieval
  • Image retrieval
  • Text representation

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

Get full text

Authors

  • Fabio CarraraConsiglio Nazionale delle Ricerche Area della Ricerca di Pisa

    Follow
  • Fabrizio FalchiIstituto di scienza e tecnologie dell'informazione Alessandro Faedo

    Follow
  • Andrea Esuli

  • Tiziano Fagni

  • Alejandro Moreo Fernández

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free