Conformance testing

Abstract

In this chapter we have presented several methods, which can uncover any fault in an implementation under different assumptions and producing checking sequences of different length and with different cost. We have initially supposed that all the assumptions of Section 4.2 hold, mainly that the machines are minimal, that the implementation does not add extra states, and that the machines have reset, status and set messages. Throughout the chapter we have presented the following methods which are capable to discover faults under a successively restricted subset of assumptions. • The method of Section 4.3, the Transition Tour (TT) method, exploits all the assumptions, except the set message. It uses a status message to check that the implementation is in the correct state. The checking sequence has length and cost linear with pn. Without a status message this method does not guarantee the detection of transfer faults. • If even a status message is not available, but the machine has still a reset message, one can use one of the methods proposed in Section 4.4, namely the W method, the Wp method, the unique input output (UIO) sequence method, the UIOv method, and the method using distinguishing sequences (DS) with reset. The DS method requires a distinguishing sequence, the UIO methods need UIOs, while W and Wp method are always applicable for minimized machines. The W, Wp, UIOv, and DS methods detect faults of any kind, while the UIO method may miss some faults. The W, Wp, and DS method with an adaptive distinguishing sequence produce checking sequences of length O(pn3) with cost O(pn3). The others have greater cost. • If even a reset message is not available, but a machine has a distinguishing sequence, the method presented in Section 4.5 uses transfer sequences instead of reset, produces checking sequences of length O(pn3) and has cost O(pn3) when used in conjunction with adaptive distinguishing sequences. • If the machine has not even a distinguishing sequence nor UIOs, the identifying sequences (IS) method, presented in Section 4.6, still works. The IS method uses only the assumptions that the implementation does not add states and that the machines are minimized and therefore they have separating sequences. It produces exponentially long checking sequences. • The problem of testing finite state machines with extra states is discussed more in general in Section 4.7, where the method originally presented by Chow [Cho78] is introduced. It is of practical interest to compare the fault detection capability of the methods when the assumptions under which they should be applied, do not hold [SL88, ZC93]. Indeed, assumptions like the equal number of states for implementation may be not verifiable in practice. The assumption of the existence of a reset message is more meaningful, but empirical studies suggest to avoid the use of the methods using reset messages for the following reason. As shown in Section 4.7, faults in extra states are more likely to be discovered when using long input sequences. The use of a reset message may prevent the implementation to reach such extra states where the faults are present. For this reason methods like UIO or DS method without reset are better in practice than the UIOv method or the DS method with reset. Although the study presented in this chapter is rather theoretical, we can draw some useful guidelines for practice testing for FSMs or for parts of models that behave like finite state machine and the reader should be aware that many ideas presented in this chapter are the basics for tools and case studies presented in Chapters 14 and 15. Such practical suggestions can improve the fault detection capability of the testing activity. • Visiting each state in a FSM (like a statement coverage) using a ST method, should not be considered enough. One should at least visit every transition using a transition tour (TT) method, that can be considered as a branch coverage. • Transition coverage should be used in conjunction of a status message to really check that the end state of every transition is the one expected. The presence of a status message in digital circuits is often required by the tester because it is of great help to uncover faults. If a status message may be not reliable, a double application of it helps to discover when it fails to reveal the correct state. • If a status message is not available (very often in software black box testing), one should use some extra inputs to verify the states. Such inputs should be unique, like in Wp, UIO and DS. • If one suspects that the implementation has more states than the implementation, he/she should prefer methods that produce long input sequences, like the DS and the IS method. However, only methods like the W method with extra states [Cho78], that add some extra inputs after visiting the transition and before checking the state identity, can guarantee to detect faults in this case. © Springer-Verlag Berlin Heidelberg 2005.