Simulation based studies in software engineering: A matter of validity

Abstract

CONTEXT: Despite the possible lack of validity when compared with other science areas, Simulation-Based Studies (SBS) performed in the context of Software Engineering (SE) have supported the achievement of some results in the field. However, as it happens with any other sort of experimental study, to increase their validity and strength confidence in the results the threats to validity must be identified and dealt. OBJECTIVE: To identify potential threats to SBS validity in SE and suggest ways to mitigate them. METHOD: To apply a secondary qualitative analysis in a dataset resulted from the aggregation of data from a quasi-systematic literature review combined with information surveyed ad-hoc regarding other science areas. RESULTS: 28 different threats to validity and concerned with SBS in SE were identified and classified according Cook and Campbell's categories. Besides, 12 verification and validation procedures applicable to SBS were analyzed to understand their ability to detect these 28 threats to valitidy. After this, the observed behaviors were used to improve guidelines regarding the planning and reporting of SBS in SE. CONCLUSIONS: Simulation based studies have different threats to validity when compared with traditional studies. They are not well known. Therefore, it is not easy to realize all of them without explicit guidance yet. So, recommendations and guidelines are presented in this paper to support their identification and mitigation.