Productive Coverage: Improving the Actionability of Code Coverage

Abstract

Code coverage is an intuitive and widely-used test adequacy measure. Established coverage measures treat each test goal (e.g., statement or branch) as equally important, and code-coverage adequacy requires every test goal to be covered. However, this is in contrast to how code coverage is used in practice. As a result, simply visualizing uncovered code is not actionable, and developers have to manually reason which uncovered code is critical to cover with tests and which code can be left untested. To make code coverage more actionable and further improve coverage in our codebase, we developed Productive Coverage - - a novel approach to code coverage that guides developers to uncovered code that should be tested by (unit) tests. Specifically, Productive Coverage identifies uncovered code that is similar to existing tested and/or frequently in production executed code. We implemented and evaluated Productive Coverage for four programming languages (C++, Java, Go, and Python), and our evaluation shows: (1) The developer sentiment, measured at the point of use, is strongly positive; (2) Productive Coverage meaningfully increases test quality, compared to a strong baseline; (3) Productive Coverage has no negative effect on code authoring efficiency; (4) Productive Coverage modestly improves code-review efficiency; (5) Productive Coverage improves code quality and prevents defects from being introduced into the code.