A diary study-based evaluation framework for mobile information retrieval

Abstract

In this poster, we propose an evaluation framework that investigates the integration of the user context (interests, location and time) into the evaluation process of mobile IR. Our approach is based on a diary study where users are asked to log their queries annotated by their location and time. Users' interests are explicitly acquired or implicitly learned based on users' relevance judgments for the retrieved documents answering their queries. We propose two evaluation protocols namely training/test in chronological order and k-fold cross validation. We exploit this framework in order to evaluate the performance of our context-based personalized mobile search approach. Experimental results show the stability performance of our approach according to the proposed evaluation protocols and demonstrate the viability of the diary approach as a means to capture context in evaluation. © 2010 Springer-Verlag.