Comparing the robustness of expansion techniques and retrieval measures

Abstract

We investigate which retrieval measures successfully discern robustness gains in the monolingual (Bulgarian, French, Hungarian, Portuguese and English) information retrieval tasks of the Ad-Hoc Track of the Cross-Language Evaluation Forum (CLEF) 2006. In all 5 of our experiments with blind feedback (a technique known to impair robustness across topics), the mean scores of the Average Precision, Geometric MAP and Precision@10 measures increased (and most of these increases were statistically significant), implying that these measures are not suitable as robust retrieval measures. In contrast, we found that measures based on just the first relevant item, such as a Generalized Success@10 measure, successfully discerned some robustness gains, particularly the robustness advantage of expanding Title queries by using the Description field instead of blind feedback. © Springer-Verlag Berlin Heidelberg 2007.