How to buy better testing: Using competition to get the most security and robustness for your dollar

Abstract

Without good testing, systems cannot be made secure or robust. Without metrics for the quality and security of system components, no guarantees can be made about the systems they are used to construct. This paper describes how firms can make the testing process faster and more cost effective while simultaneously providing a reliable metric of quality as one of the outputs of the process. This is accomplished via a market for defect reports, in which testers maximize profits by minimizing the cost of finding defects. The power of competition is harnessed to ensure that testers are paid a fair price for the defects they discover, thereby aligning their incentives with those of the firm developing the system. The price to find, demonstrate, and report a defect that is set by the market serves as the measure of quality.