Automated component-level evaluation: Present and future

Abstract

Automated component-level evaluation of information retrieval (IR) is the main focus of this paper. We present a review of the current state of web-based and component-level evaluation. Based on these systems, propositions are made for a comprehensive framework for web service-based component-level IR system evaluation. The advantages of such an approach are considered, as well as the requirements for implementing it. Acceptance of such systems by researchers who develop components and systems is crucial for having an impact and requires that a clear benefit is demonstrated. © 2010 Springer-Verlag Berlin Heidelberg.