Supporting Peer-to-Peer Learning with LLMs: Investigating Smarter Student Solution Recommendations

Abstract

Peers can offer valuable advice, but effectively matching students with relevant peer recommendations remains challenging. Prior research proposed a reflection recommender system that shares students’ responses to reflection questions about their challenges in a course with peers who shared similar challenges. We incorporated a large language model (LLM) to enhance the system’s ability to generate personalized responses, filter out irrelevant reflections, and provide relevant advice for students’ challenges. We also formatted the output as a conversational response, in contrast to the unmodified list of student solutions. We compare the original student-challenge recommender system (SCS) to our LLM-integrated student-challenge recommender system (LLM-SCS) in a comparative study across three computer science courses, using A/B testing to mitigate order effects. We asked students to rate solutions from both systems on a 7-point Likert scale and provide free-response feedback. A t-test demonstrated that the differences were not statistically significant, demonstrating that students find the quality of LLM-SCS and SCS responses comparable. An investigation of free-response feedback reveals that students express diverse needs and preferences; some preferred the conversational style of the LLM-SCS response, and others valued the uniquely tailored advice of the original response. In this work, we explore these differences in great depth.