Microsoft’s Protocol Documentation Program: A Success Story for Model-Based Testing

Abstract

Microsoft is producing interoperability documentation for Windows client-server and server-server protocols. The Winterop team in the Windows organization is responsible for verifying the documentation to ensure its quality. Various test-driven methods are being applied including, when appropriate, a model- based approach. This talk describes core aspects of the quality assurance process and tools that were put in place, and specifically focuses on model-based testing (MBT), using Microsoft’s Spec Explorer technology. Though MBT has been applied successfully to features and products before, this is the first attempt to use it in such a large scale and in the context of a business-critical area within Microsoft, and to the best of the author’s knowledge throughout the whole industry. Empirical results confirm that MBT not only works and that it scales, provided it is accompanied by good tool support and clear methodological guidance, but also that it delivers significant productivity gains when compared to traditional testing in a similar domain and with the same people resources. The method to measure this is based on the effort per end-to-end tested requirement (including not only test design, but also test harnessing and test execution), over a probe of 9,844 requirements in test suites using MBT and 8,728 requirements in non-MBT test suites, with an average effort of 1.39 person days for MBT-tested requirement, and 2.37 person days for traditionally-tested requirement, documenting a productivity gain of 42%. A detailed account of this work is scheduled to appear in the journal for Software Testing and Verification (STVR) under the title Model-based Quality Assurance of Protocol Documentation: Tools and Methodology.