Recipe Retrieval with Visual Query of Ingredients

Abstract

Recipe retrieval is a representative and useful application of cross-modal information retrieval. Recent studies have proposed frameworks for retrieving images of cuisines given textual ingredient lists and instructions. However, the textual form of ingredients easily causes information loss or inaccurate description, especially for novices of cookery who are often the main users of recipe retrieval systems. In this paper, we revisit the task of recipe retrieval by taking images of ingredients as input queries, and retrieving cuisine images by incorporating visual information of ingredients through a deep convolutional neural network. We build an image-to-image recipe retrieval system to validate the effect of ingredient image queries. We further combine the proposed solution with a state-of-the-art cross-modal recipe retrieval model to improve the overall performance of the recipe retrieval task.