Excellence in Exploratory Testing: Success Factors in Large-Scale Industry Projects

Abstract

Based on interviews with 20 interviewees from four case study companies, this paper presents a list of key factors that enable efficient and effective exploratory testing of large-scale software systems. The nine factors are grouped into four themes: “The testers’ knowledge, experience and personality”, “Purpose and scope”, “Ways of working” and “Recording and reporting”. According to the interviewees, exploratory testing is a more creative way to work for the testers, and was therefore considered to make better use of the testers. Exploratory testing was also described as a good way to test system-wide and to test large-scale systems, especially exploratory testing with an end-user perspective. The identified key factors were confirmed by a series of follow-up interviews with the 20 interviewees and a cross-company workshop with 14 participants. This strengthens the generalizability of the findings, supporting that the list of key factors can be applied to projects in a large segment of the software industry. This paper also presents the results from a systematic literature review including 129 publications related to exploratory testing. No publication were found that summarizes the key factors that enable efficient and effective exploratory testing, which supports the novelty of the findings.