Intelligent evaluation of test suites for developing efficient and reliable software

Abstract

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.