Mutation-Based White Box Testing of Deep Neural Networks

Abstract

Deep Neural Networks (DNNs) are used in many critical areas, such as autonomous vehicles, generative AI systems, etc. Therefore, testing DNNs is vital, especially for models used in critical areas. Mutation-based testing is a very successful technique for testing DNNs by mutating their complex structures. Deep Mutation Module was developed to address mutation-based testing and the robustness challenges of DNNs. It analyses the structures of DNNs in detail. It tests models by applying mutation to parameters and structures using its fault library. Testing DNN structures and detecting faults is a highly complex and open-ended challenge. The method proposed in this study applies mutations to DNN parameters to expose faults and weaknesses in the models, thereby testing their robustness. The paper focuses on mutation-based tests of an Reinforce Learning (RL) model developed for electric vehicle routing, a Long Short-Term Memory (LSTM) model developed for prognostic predictions, and a Transformer-based neural network model for electric vehicle routing tasks. The best mutation scores for the LSTM model were measured as 96%, 91.02%, 71.19%, and 68.77%. The test results for the RL model resulted in mutation scores of 93.20%, 72.13%, 77.47%, 79.28%, and 55.74%. The mutation scores of the Transformer model were 75.87%, 76.36%, and 74.93%. These results show that the module can successfully test the targeted models and generate mutants classified as 'survived mutants' that outperform the original models. In this way, it provides critical information to researchers to improve the overall performance of the models. Conducting these tests before using them in real-world applications minimizes faults and maximizes model success.