Model-based testing areas, tools and challenges: A tertiary study

Abstract

Context: Model-based testing is one of the most studied approaches by secondary studies in the area of software testing. Aggregating knowledge from secondary studies on model-based testing can be useful for both academia and industry. Objective: The goal of this study is to characterize secondary studies in model-based testing, in terms of the areas, tools and challenges they have investigated. Method: We conducted a tertiary study following the guidelines for systematic mapping studies. Our mapping included 22 secondary studies, of which 12 were literature surveys and 10 systematic reviews, over the period 1996–2016. Results: A 3-level hierarchy of model-based testing areas was built based on existing taxonomies as well as data that emerged from the secondary studies themselves. This hierarchy was then used to classify the studies, tools, challenges and their tendencies in a unified classification scheme. The most studied areas overall are two model paradigms: UML models and transition-based notations, and two test levels: unit and integration testing. Only five studies were found to compare and classify model-based testing tools, which motivated us to classify all tools found into our hierarchy of areas. Most tools fell within the model paradigm area. Over time, tools that test the functional behavior have prevailed, with a recent tendency to support executable tests. With regard to model-based testing challenges, most of them are associated to the model specification area. Besides, a grounded analysis was done on challenges, yielding six categories. Availability is the category where more challenges are reported. Over time, challenges have moved from complexity to lack of approaches for specific software domains. Conclusions: Only a few systematic reviews on model-based testing could be found, therefore some areas still lack secondary studies, particularly, test execution aspects, language types, model dynamics, as well as some model paradigms and generation methods. We thus encourage the community to perform further systematic reviews and mapping studies, following known protocols and reporting procedures, in order to increase the quality and quantity of empirical studies in model-based testing.