Benchmarking Large Language Models for Autonomous Run-time Error Repair: Toward Self-Healing Software Systems

Abstract

As software systems grow in complexity and become integral to daily operations, traditional approaches to software testing, maintenance, and evolution are increasingly inadequate. Recent advances in artificial intelligence, particularly in large language models, offer promising avenues for achieving self-healing software - software capable of autonomously detecting, diagnosing, and repairing faults without human intervention. However, while much of the existing literature focuses on on code repair of vulnerabilities or repository-level bugs, the application of large language models for autonomously repairing run-time errors - which require dynamic analysis and execution context awareness - remains largely uncharted.In this study, we empirically benchmark ten distinct large language models - ChatGPT-4o, ChatGPT-4o-mini, Claude 3.5 Sonnet, Claude 3.5 Haiku, Gemini 1.5 Flash, Llama 3.2, Mistral Nemo, Grok Beta, Command R+, and Jamba 1.5 Large - to assess their ability to repair run-time errors in code. We conducted our evaluation on a dataset of 76 programming problems manually sourced from Leetcode, implemented in C++ (48 problems) and Java (28 problems). Each model was provided with a single opportunity to generate a corrected solution, which was then evaluated based on its ability to pass all associated test cases.Our experimental results provide early empirical evidence of the potential of large language models to drive a paradigm shift in artificial intelligence-driven software engineering. The findings reveal that while certain large language models demonstrate strong code-fixing capabilities, others struggle, highlighting significant performance disparities across models. This work not only fills a critical gap in empirical software engineering, but also opens avenues for refining artificial intelligence-driven software engineering, particularly for self-healing software.