Coping with sparsity in a recommender system

Abstract

In this paper we report experiments that we conducted using an implementation of a recommender system called "Knowledge Pump" (KP) developed at Xerox. We repeat well-known methods such as the Pearson method, but also address common problems of recommender systems, in particular the sparsity problem. The sparsity problem is the problem of having too few ratings and hence too few correlations between users. We address this problem in two different manners. First, we introduce "transitive correlations", a mechanism to increase the number of correlations between existing users. Second, we add "agents", artificial users that rate in accordance with some predefined preferences. We show that both ideas pay off, albeit in different ways: Transitive correlations provide a small help for virtually no price, whereas rating agents improve the coverage of the system significantly but also have a negative impact on the system performance. © Springer-Verlag Berlin Heidelberg 2003.