Automated Feedback Generation for Competition-Level Code

Abstract

Competitive programming has become a popular way for programmers to test their skills. Competition-level programming problems are challenging in nature, and participants often fail to solve the problem on their first attempt. Some online platforms for competitive programming allow programmers to practice on competition-level problems, and the standard feedback for an incorrect practice submission is the first test case that the submission fails. Often, the failed test case does not provide programmers with enough information to resolve the errors in their code, and they abandon the problem after making several more unsuccessful attempts. We present Clef, the first data-driven tool that can generate feedback on competition-level code automatically by repairing programmers' incorrect submissions. The key development is that Clef can learn how to generate repairs for incorrect submissions by examining the repairs that other programmers made to their own submissions over time. Since the differences between an incorrect program and a correct program for the same task may be significant, we introduce a new data structure, merge trees, to capture the changes between submissions. Merge trees are versatile: they can encode both large algorithm-level redesigns and small statement-level alterations. We evaluated Clef on six real-world problems from Codeforces, the world's largest platform for competitive programming. Clef achieves accuracy in repairing programmers' incorrect submissions. When given incorrect submissions from programmers who never found the solution to a problem on their own, Clef repairs the users' programs of the time.