Choosing an ideal recall-base for the evaluation of the focused task: sensitivity analysis of the XCG evaluation measures

Abstract

The paper investigates the measures of retrieval effectiveness employed in the ad-hoc track of INEX 2006. In particular, it looks at how the evaluation of the Focused task is affected when different methodology is employed in generating the so-called ideal recall-base, which forms the ground-truth of the evaluation. The results show that the choice of methodology can impact on the obtained performance scores and the relative ranking of systems in relation to each other, especially when the effectiveness scores are uniformly low across all systems. Most XCG measures show very similar levels of sensitivity to changes in the ideal recall-base. © Springer-Verlag Berlin Heidelberg 2007.