Improving the Efficacy of Testing Scientific Software: Insights from Mutation Testing

Abstract

Scientific software is often used for critical decision-making in various fields, such as chemistry, physics, biology, medicine, and earth science. Due to the complexity and domain knowledge required to create this software, the end users, i.e. scientists, often develop them themselves. Consequently, one of the areas that has received less attention in scientific software development is evaluating the test quality. One of the common approaches used to assess the test quality is mutation testing. However, mutation testing is rarely used in scientific software development due to resource constraints. Thus, in this study, we conduct mutation testing on several scientific software projects to learn how the quality of tests in scientific software can be improved based on the outcomes of mutation testing. Our results show that these scientific software projects have some common test deficiencies across them, and some of these deficiencies can be rectified using some well-established testing techniques and improving the testing knowledge of the scientific software developers.