A study on novelty evaluation in biomedical information retrieval

Abstract

In novelty information retrieval, we expect that novel passages are ranked higher than redundant ones and relevant ones higher than irrelevant ones. Accordingly, we desire an evaluation algorithm that would respect such expectations. In TREC 2006 & 2007, a novelty performance measure, called the aspect-based mean average precision (MAP), was introduced to the Genomics Track to rank the novelty of the medical passages. In this paper, we demonstrate that this measure may not necessarily yeild a higher score for the rankings that honor above expectations better. We propose an improved measure to reflect such expectations more precisely, and present some supporting evidences. © 2012 Springer-Verlag Berlin Heidelberg.