Planning to Guide LLM for Code Coverage Prediction

Abstract

Code coverage serves as a crucial metric to assess testing effectiveness, measuring the degree to which a test suite exercises different facets of the code, such as statements, branches, or paths. Despite its significance, coverage profilers necessitate access to the entire codebase, constraining their usefulness in situations where the code is incomplete or execution is not feasible, and even cost-prohibitive. In this paper, we present CodePilot, a plan-based prompting approach grounded in program semantics, which collaborates with a Large Language Model (LLM) to enhance code coverage prediction. To address the intricacies of predicting code coverage, CodePilot employs planning by discerning various types of statements in an execution flow. Planning empowers GPT to autonomously generate plans based on guided examples, and then CodePilot prompts the GPT model to predict code coverage (Action) based on the plan it generated (Reasoning). Our experiments evaluating CodePilot demonstrate high accuracy, achieving up to 55% in exact-match and 89% in statement-match. It performs relatively better than the baselines, achieving up to 33% and 19% relatively higher in those metrics. We also showed that due to highly accurate plans (90%), GPT model predicts better code coverage. Moreover, we show CodePilot's utility in correctly predicting the least covered statements.