Achieving high performance on sketch-based image retrieval without real sketches for training

Abstract

Sketch-based image retrieval (SBIR) has become an attractive area in computer vision. Along with the advances in deep learning, we have seen more sophisticated models for SBIR that have shown increasingly better results. However, these models are still based on supervised learning strategies, requiring the availability of real sketch-photo pairs. Having a paired dataset is impractical in real environments (e.g. eCommerce), which can limit the massification of this technology. Therefore, based on advances in foundation models for extracting highly semantic features from images, we propose S3BIR-DINOv2, a self-supervised SBIR model using pseudo-sketches to address the absence of real sketches for training, learnable vectors to allow the model to hold only one encoder for processing the underlying two image modalities, contrastive learning and an adapted DINOv2 as the visual encoder. Our experiments show our model performs outstandingly in diverse public datasets without requiring real sketches for training. We reach an overall mAP of 61.10% in Flickr15K and 44.37% in the eCommerce dataset.