Mutation-based test suite evolution for self-organizing systems

Abstract

We consider test design as an optimization problem. The challenge is to find a set of test cases, the so-called test suite, that optimizes two quantifiable requirements: First, the effort needed for test execution should be minimal; a given test budget usually sets a maximum for the size of the test suite. Second, the test suite should maximize the score of a given test goal estimating its error detection capability, e.g., by the use of coverage or risk metrics. This paper studies test design for testing self-organizing systems with a mutation-based test goal. Equipped with a reconfiguration mechanism, this kind of a distributed system adapts its internal structure and thus its behavior to changing environmental conditions at run time. Test execution at a time step t consequently not only triggers an observable output at t + 1, but might also bring about a reconfiguration of the system under test influencing the result of subsequently executed test cases. Formalizing the evolving sequential decision problem of test case executions by dependency graphs, in which we try to find optimal sets of paths for the mutation-based goal, we investigate the suitability of various kinds of evolutionary algorithms for optimization. All of the considered algorithms are evaluated using a concrete case study of an adaptive, self-organizing production cell.