User-centered evaluation for IR: Ranking annotated document algorithms

Abstract

Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document. © 2011 Springer-Verlag.