A brief index for proximity searching

Abstract

Many pattern recognition tasks can be modeled as proximity searching. Here the common task is to quickly find all the elements close to a given query without sequentially scanning a very large database. A recent shift in the searching paradigm has been established by using permutations instead of distances to predict proximity. Every object in the database record how the set of reference objects (the permutants) is seen, i.e. only the relative positions are used. When a query arrives the relative displacements in the permutants between the query and a particular object is measured. This approach turned out to be the most efficient and scalable, at the expense of loosing recall in the answers. The permutation of every object is represented with κ short integers in practice, producing bulky indexes of 16κn bits. In this paper we show how to represent the permutation as a binary vector, using just one bit for each permutant (instead of log κ in the plain representation). The Hamming distance in the binary signature is used then to predict proximity between objects in the database.We tested this approach with many real life metric databases obtaining faster queries with a recall close to the Spearman ρ using 16 times less space. © 2009 Springer-Verlag Berlin Heidelberg.