An approach for the construction of an experimental test collection to evaluate search systems that exploit annotations

Abstract

This study addresses the lack of an adequate test collection that can be used to evaluate search systems that exploit annotations to increase the retrieval effectiveness of an information search tool. In particular, a new approach is proposed that enables the automatic creation of multiple test collections without human effort. This approach takes advantage of the human relevance assessments contained in an already existing test collection and it introduces content-level annotations in that collection. © Springer-Verlag Berlin Heidelberg 2007.