Navigating the Feedback Loop in Recommender Systems: Insights and Strategies from Industry Practice

Abstract

Understanding and measuring the impact of feedback loops in industrial recommender systems is challenging, leading to the underestimation of their deterioration. In this study, we define open and closed feedback loops and investigate the unique reasons behind the emergence of feedback loops in the industry, drawing from real-world examples that have received limited attention in prior research. We highlight the measurement challenges associated with capturing the full impact of feedback loops using traditional online A/B tests. To address this, we propose the use of offline evaluation frameworks as surrogates for long-term feedback loop bias, supported by a practical simulation system using real data. Our findings provide valuable insights for optimizing the performance of recommender systems operating under feedback loop conditions.