Test Input Selection for Deep Neural Networks

Abstract

With the rapid development of deep learning technologies, the quality and security of deep learning systems have aroused great concern recently. Much research has been done in testing deep learning systems. Nevertheless, the oracle problem still remains since the input space of DNN-based software is usually very large and manually labeling is boring and cost-effective. Inspired by structural testing for traditional software, in this paper, we propose an algorithm to mitigate the oracle problem by selecting test inputs worth labeling. Specifically, we propose a test subset selection algorithm that can automatically select a test suite with high coverage but a small size when the labeling budget is limited. Compared with DeepXplore, the proposed algorithm can generate smaller test suites with higher coverage.