Test oracles

Abstract

All software testing methods depend on the availability of an oracle, that is, some method for checking whether the system under test has behaved correctly on a particular execution. An ideal oracle would provide an unerring pass/fail judgment for any possible program execution, judged against a natural specification of intended behavior. Practical approaches must make compromises to balance trade-offs and provide useful capabilities. This report surveys proposed approaches to the oracle problem that are general in the sense that they require neither pre-computed input/output pairs nor a previous version of the system under test. The survey is not encyclopedic, but discusses representative examples of the main approaches and tactics for solving common problems.