Using experimental material management tools in experimental replication: A systematic mapping study

Abstract

Context: In experimental software engineering (ESE), experimental replication is applied to validate results of an experiment. Much information and one version of the experimental materials are required for replication. Prior to the replication execution, all or part of the materials may require changes, producing new or modified versions of these, which should be incorporated into the material of the original experiment. There is a direct relationship between the increase in the number of replications with the increase of versions of the experimental material, which commonly causes confusion and disorder for experimental administrators. Objective: The aim of this paper is to conduct a mapping study to locate articles about the use of experimental material management tools in experimental replication in ESE. Method: We applied the mapping study to search, analyze and select published papers from reported replications. Results: We analyzed a total of 592 articles published from 1998 to 2014, 24 of them have been pre-selected and 4 have finally been selected. Conclusion: The results show the limited existence of articles on this subject. In addition, our analysis has allowed to identify that most of them suffer from problems in versions management for both replication and experimental material. These data provide information of interest to start a research about adoption of the paradigm of software configuration management inside the management of the experimental material in ESE.