Formal specifications and test: Correctness and oracle

Abstract

This article presents a new formal approach to testing. In the field of dynamic testing, as soon as a program fails for a test set, it is flagged incorrect. The remaining question is: how far can a successful program be considered as correct? We give a definition of program correctness with respect to a specification which is adequate to dynamic testing. Similarly to the field of abstract implementation, the idea is that in order to declare a program as correct, it suffices that its behavior fulfills the specification requirements. An intermediate semantic level between the program and the specification, called the oracle framework, is introduced in order to interpret observable results obtained from dynamic experiments on the program. This allows to give algebraic semantics (i.e. a set of models) to the program, compatible with the program behavior. Program correctness is then defined by some adequacy criterion between the specification semantics and the program semantics. We point out that while for some specifications, there exist exhaustive test sets (the success of which means program correctness), for some other specifications, there only exist "complete" (but not exhaustive) test sets. Of course, all the programs rejected by a complete test set are incorrect but unfortunately, there still exist successful incorrect programs. We also explain how the test set selection can be formalized within our approach.