Comparative recommender system evaluation

  • Said A
  • Bellogín A
  • 141


    Mendeley users who have this article in their library.
  • 41


    Citations of this article.


Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from dif-ferent recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithmic implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To pro-vide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magni-tude better or worse across frameworks. Our results show the neces-sity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Alan Said

  • Alejandro Bellogín

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free