Continuous Experimentation for Software Organizations with Low Control of Roadmap and a Large Distance to Users: An Exploratory Case Study

Abstract

With the increasing popularity of A/B testing and other experimentation practices in web systems, companies from a range of different domains are starting to look at continuous experimentation as a way to guide product development and feature prioritization. Research in continuous experimentation traditionally focused on companies that have easy access to user data and that have a high degree of control of the product roadmap. However, little research has been conducted to understand how companies that have a low control of roadmap and have a large distance to the users, such as consultancy companies, can benefit from continuous experimentation practices. To address this problem, we performed an exploratory case study with a software consultancy company combined with a validation procedure with four additional companies. The contribution of this work is three-fold. First, we devised a model to classify a company in the terms of the distance to users and the control of roadmap. Second, we show how control of roadmap and distance to user impacts continuous experimentation. Finally, we present several perceived challenges and benefits of continuous experimentation for companies and directions for future work.