Atribute-based evaluation for recommender systems: Incorporating user and item atributes in evaluation metrics

Abstract

Research in Recommender Systems evaluation remains critical to study the efciency of developed algorithms. Even if diferent aspects have been addressed and some of its shortcomings - such as biases, robustness, or cold start - have been analyzed and solutions or guidelines have been proposed, there are still some gaps that need to be further investigated. At the same time, the increasing amount of data collected by most recommender systems allows to gather valuable information from users and items which is being neglected by classical ofine evaluation metrics. In this work, we integrate such information into the evaluation process in two complementary ways: on the one hand, we aggregate any evaluation metric according to the groups defned by the user attributes, and, on the other hand, we exploit item attributes to consider some recommended items as surrogates of those interacted by the user, with a proper penalization. Our results evidence that this novel evaluation methodology allows to capture diferent nuances of the algorithms performance, inherent biases in the data, and even fairness of the recommendations.