Techniques for testing scientific programs without an oracle

  • Kanewala U
  • Bieman J
  • 43


    Mendeley users who have this article in their library.
  • 13


    Citations of this article.


The existence of an oracle is often assumed in software testing. But in many situations, especially for scientific programs, oracles do not exist or they are too hard to implement. This paper examines three techniques that are used to test programs without oracles: (1) Metamorphic testing, (2) Run-time Assertions and (3) Developing test oracles using machine learning. We examine these methods in terms of their (1) fault finding ability, (2) automation, and (3) required domain knowledge. Several case studies apply these three techniques to effectively test scientific programs that do not have oracles. Certain techniques have reported a better fault finding ability than the others when testing specific programs. Finally, there is potential to increase the level of automation of these techniques, thereby reducing the required level of domain knowledge. Techniques that can potentially be automated include (1) detection of likely metamorphic relations, (2) static analyses to eliminate spurious invariants and (3) structural analyses to develop machine learning generated oracles.

Author-supplied keywords

  • Assertion checking
  • Machine learning
  • Metamorphic relation
  • Metamorphic testing
  • Mutation analysis
  • Scientific software testing
  • Test oracles

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • Upulee Kanewala

  • James M. Bieman

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free