Testing with guarantees and the failure of regression testing in eXtreme programming

Abstract

The eXtreme Programming (XP) method eschews all formal design, but compensates for this by rigorous unit testing. Test-sets, which constitute the only enduring specification, are intuitively developed and so may not be complete. This paper presents a method for generating complete unit test-sets for objects, based on simple finite state machines. Using this method, it is possible to prove that saved regression test-sets do not provide the expected guarantees of correctness when applied to modified or extended objects. Such objects, which pass the saved tests, may yet contain introduced faults. This puts the whole practice of regression testing in XP into question. To obtain the same level of guarantee, tests must be regenerated from scratch for the extended object. A notion of guaranteed, repeatable quality after testing is defined. © Springer-Verlag Berlin Heidelberg 2005.