Query based object retrieval using neural codes

Abstract

The task of retrieving a specific object from an image, which is similar to a query object is one of the critical applications in the computer vision domain. The existing methods fail to return similar objects when the region of interest is not specified correctly in a query image. Furthermore, when the feature vector is large, the retrieval from big collections is usually computationally expensive. In this paper, we propose an object retrieval method, which is based on the neural codes (activations) generated by the last inner-product layer of the Faster R-CNN network demonstrating that it can be used not only for object detection but for retrieval too. To evaluate the method, we have used a subset of ImageNet comprising of images related to indoor scenes, and to speed-up the retrieval, we first process all the images from the dataset and we save information (i.e. neural codes, objects present in the image, confidence scores and bounding box coordinates) corresponding to each detected object. Then, given a query image, the system detects the object present and retrieves its neural codes, which are then used to compute the cosine similarity against saved neural codes. We retrieved objects with high cosine similarity scores, and then we compared it with the results obtained using confidence scores. We showed that our approach takes only 0.534 s to retrieve all the 1454 objects in our test set.