Does Live Regression Testing Help?

Abstract

Regression testing is an expensive, yet crucial part of the software development process. As regression test suites grow in size, the time required for their execution increases proportionally, and their execution is often either delegated to a specialized testing environment out of developers reach, or they are omitted completely. This could have a variety of negative effects on the developers’ productivity, including interruptions and slowdown of developers’ workflow. We propose a method of live regression unit testing to address these issues via incorporating Regression Test Selection and Test Case Prioritization techniques and an automatized change detection mechanism to run the regression testing in the background automatically. By combining the test results with source code changes and code coverage information, we are able to precisely identify source code changes responsible for test failures. By the paired two-sample t-test we proved, that our method is able to increase the speed of fault detection and to fix changes responsible for incorrect behaviour almost 2 times (p-value = 0.001, = 0.05).