Parameterized decay model for information retrieval

Abstract

This article proposes a term weighting scheme for measuring query-document similarity that attempts to explicitly model the dependency between separate occurrences of a term in a document. The assumption is that, if a term appears once in a document, it is more likely to appear again in the same document. Thus, as the term appears again and again, the information content of the subsequent occurrences decreases gradually, since they are more predictable. We introduce a parameterized decay function to model this assumption, where the initial contribution of the term can be determined using any reasonable term discrimination factor. The effectiveness of the proposed model is evaluated on a number of recent web test collections of varying nature. The experimental results show that the proposed model significantly outperforms a number of well known retrieval models including a recently proposed strong Term Frequency and Inverse Document Frequency (TF-IDF) model.