Prioritizing Metamorphic Relations via Execution Profile Dissimilarity for Improved Fault Detection

Abstract

An oracle is a mechanism that determines whether the output of the program for the executed test cases is correct. For machine learning programs, such an oracle is not available or is too difficult to apply. Metamorphic testing is a testing approach that uses metamorphic relations, which are necessary properties of the software under test to help verify the correctness of a program. Prioritization of metamorphic relations helps to increase fault detection effectiveness and improve metamorphic testing efficiency. However, prioritizing metamorphic relations based on faults and code coverage is often not effective for prioritizing MRs since fault-based ordering of MRs is expensive and the code coverage-based approach can provide inaccurate ordering of MRs. To this end, in this work, we propose an approach based on diversity in the execution profile of the source and follow-up test cases to prioritize metamorphic relations. We show that the proposed statement centrality-based prioritization approach increases the effectiveness of fault detection by up to 31% compared to the code coverage-based approach and reduces the time taken to detect a fault by 29% compared to random execution of MR. In general, our approach led to an increase in the average rate of fault detection, reduced the time taken to detect a fault, and increased fault detection effectiveness.