Adapting a spatial access structure for document representations in vector space

Abstract

In the field of information-retrieval the vector space model has been proposed. In this model queries and documents are represented as term vectors where each coefficient represents the relevance of a given term with respect to the document or query. A typical task in this context is to search for the documents most similar to a given query vector. On the other hand, algorithms to perform nearest neighbor and distance scan queries have been proposed for various types of spatial access structures. Unfortunately, these access structures assume implicitly that the number of dimensions is relatively small - which is not the case for document representation vectors. In this paper we discuss the adaptation of spatial access structures for document representation vectors. We describe how some peculiarities of document representation vectors can be exploited to overcome the problems with higher dimensions to a certain extend. We exploit these peculiarities introducing a new cluster split technique and a sophisticated algorithm to calculate an upper bound for the similarity of the documents located in a subtree of the access structure.