Content-Based Image Retrieval Using Hand-Drawn Sketches and Local Features: a Study on Visual Dissimilarity

Abstract

This thesis addresses the question of content-based image retrieval (CBIR) in heterogeneous databases. In an analysis of the existing CBIR tools that was done at the beginning of this work, we have shown that there was room for improvement in three key areas: query form, image and query representation, and computation of similarity. This analysis led us to studying the usability of a method for computing dissimilarity between user-produced pictorial queries and database images according to features extracted from automatically segmented homogeneous areas. The proposed approach differentiates itself from the analyzed ones by giving maximum freedom to the user by using user-produced pictorial queries (sketches) depicting the wanted image(s), extracts visual information from areas of the images automatically recognized as visually homogeneous and allows the comparison of database images with queries containing various levels of detail, thanks to a hierarchical representation of both database images and queries. Sketches can be incomplete (i.e., they do not need to cover all the available canvas), resulting in extra flexibility. Furthermore, the method can be combined with classical CBIR methods, such as keyword indexing. In order to support our proposal, a prototype CBIR system, SimEstIm, was built. In SimEstIm, the user produces a query image with a paint tool, then submits it to the system, which extracts a query representation. At database population time, database images undergo the same treatment, which consists of two steps: region segmentation and region merging. In order to allow the comparison between database images and sketches containing various levels of detail, several segmentation results are stored for each image. Visual dissimilarity is computed as a combination of dissimilarities between the regions in the query and the regions in the database image's segmentation results, resulting in dissimilarity values for each segmentation result. These results are then used to compute a unique dissimilarity score between query and image. The user can control the behavior of the dissimilarity measure by setting weights associated to each visual feature. Experiments were performed by several users. The results obtained are extremely encouraging, and show that the proposed method can be successfully implemented in a CBIR system.