8 1/2

Abstract

The high cost and difficulty of realistic benchmarks encourages most computer purchasers to use informal performance validation methods during acquisitions. A variety of performance data is available - rating charts, references from current users, standardized benchmark results, modeling, and so on. There are also a variety of structured and unstructured approaches to evaluating this data. Experience shows that virtually all of this data is flawed. This paper summarizes the experiences of several federal agencies that have experimented with informal performance validation methods. It points out problems encountered, and suggests structured methods for responding to these problems.