Exploring Prompt Patterns for Effective Vulnerability Repair in Real-World Code by Large Language Models

Abstract

Large Language Models (LLMs) have shown promise in automating code vulnerability repair, but their effectiveness in handling real-world code remains limited. This paper investigates the capability of LLMs, in repairing vulnerabilities and proposes a systematic approach to enhance their performance through specialized prompt engineering. Through extensive evaluation of 5,826 code samples, we found that while LLMs successfully repair vulnerabilities in simple cases, they struggle with complex real-world code that involves intricate dependencies, contextual requirements, and multi-file interactions. To address these limitations, we first incorporated Control Flow Graphs (CFGs) as supplementary prompts, achieving a 14.4% success rate in fixing previously unresolvable vulnerabilities. Through analysis of repair failures, we identified three primary challenge categories and developed corresponding prompt patterns incorporating techniques such as granular contextual information provision and progressive code simplification. Evaluation on real-world projects demonstrated that our approach significantly improved LLMs’ repair capabilities, achieving over 85% success rates across all identified challenge categories. Our findings suggest that while LLMs have inherent limitations in handling complex vulnerabilities independently, they can become effective tools for automated vulnerability repair when guided by carefully crafted prompts. Due to constraints imposed by one of the NIST authors, we are unable to disclose the specific LLMs that have been tested.