Dataset Diversity for Metamorphic Testing of Machine Learning Software

Abstract

Machine learning software is non-testable in that training results are not available in advance. The metamorphic testing, using pseudo oracle, is promising for software testing of such machine learning programs. Machine learning software, indeed, works on a collection of a large number of data, and thus slight changes in the input training dataset have a large impact on training results. This paper proposes a new metamorphic testing method applicable to neural network learning models. Key ideas are dataset diversity as well as behavioral oracle. Dataset diversity takes into account the dataset dependency of training results, and provides a new way of generating follow-up test inputs. Behavioral oracle monitors changes of certain statistical indicators as training processes proceed and is a basis of metamorphic relations to be checked. The proposed method is illustrated with a case of software testing of neural network programs to classify handwritten numbers.