User Story Quality in Practice: A Case Study

Abstract

(1) Background: User stories are widely used in Agile development as requirements. However, few studies have assessed the quality of user stories in practice. (2) Methods: What is the quality of user stories in practice? To answer the research question, we conducted a case study. We used an analysis report from a real-life project where an organization wanted to improve its existing hotline system or acquire a new one. We invited IT practitioners to write requirements for the new system based on the analysis report, user stories, and whatever else they considered necessary. The practitioners could ask the authors questions as they would ask a customer in a real setting. We evaluated the practitioners’ replies using these IEEE 830 quality criteria: completeness, correctness, verifiability, and traceability. (3) Results: The replies covered only 33% of the needs and wishes in the analysis report. Further, the replies largely missed other requirements needed in most projects, such as learnability and maintainability. Incorrect or restrictive solutions were often proposed by the practitioners. Most replies included user stories that were hard to verify, or would have caused a cumbersome user interface if implemented independently. (4) Conclusion: In this project, relying on the user stories would have been a disaster. Although the user stories could have been improved, they wouldn’t cover the necessary requirements in the project.