Matching Search Result Diversity with User Diversity Acceptance in Web Search Sessions

Abstract

Promoting diversity in ranking while maintaining the relevance of ranked results is critical for enhancing human-centered search systems. While existing ranking algorithm and diversity IR metrics provide a solid basis for evaluating and improving search result diversification in offline experiments, it misses out possible divergences and temporal changes of users' levels of Diversity Acceptance, which in this work refers to the extent to which users actually prefer to interact with topically diversified search results. To address this gap between offline evaluations and users' expectations, we proposed an intuitive diversity acceptance measure and ran experiments for diversity acceptance prediction and diversity-aware re-ranking based on datasets from both controlled lab and naturalistic settings. Our results demonstrate that: 1) user diversity acceptance change across different query segments and session contexts, and can be predicted from search interaction signals; 2) our diversity-aware re-ranking algorithm utilizing predicted diversity acceptance and estimated relevance labels can effectively minimize the gap between diversity acceptance and result diversity, while maintaining SERP relevance levels. Our research presents an initial attempt on balancing user needs, result diversity, and SERP relevance in sessions and highlights the importance of studying diversity acceptance in promoting effective result diversification.