An effective recall-oriented information retrieval system evaluation

Abstract

The objective recall-oriented retrieval tasks such as patent retrieval, legal search, and e-Discovery is to find all relevant documents. These information retrieval tasks are usually conducted by the domain experts so the retrieval cost might be high. However, the existing evaluation metrics for recall-oriented retrieval tasks have obvious limitations which consider only high recall value. If the relevant documents are at the lower position in the total ranked result, the user should check the whole result up to the end. Therefore, the evaluation measure for recall-oriented retrieval tasks is strongly required reducing the review efforts. In this paper, we study the feature of the various evaluation metrics according to the ranks of the query result in the recall-oriented retrieval tasks.