Weighted rank correlation in information retrieval evaluation

Abstract

In Information Retrieval (IR), it is common practice to compare the rankings observed during an experiment - the statistical procedure to compare rankings is called rank correlation. Rank correlation helps decide the success of new systems, models and techniques. To measure rank correlation, the most used coefficient is Kendall's τ. However, in IR, when computing the correlations, the most relevant, useful or interesting items should often be considered more important than the least important items. Despite its simplicity and widespread use, Kendall's τ little helps discriminate the items by importance. To overcome this drawback, in this paper, a family τ * of rank correlation coefficients for IR has been introduced for discriminating the rank correlation according to the rank of the items. The basis has been provided by the notion of gain previously utilized in retrieval effectiveness measurement. The probability distribution for τ * has also been provided. © 2009 Springer Berlin Heidelberg.