A methodology for objectively evaluating error messages

Abstract

Message quality is a critical factor in influencing user acceptance of a program product. Good error messages can reduce the time and cost to create and maintain software, as well as help users learn about the product. We have developed a methodology for conducting controlled usability evaluations of error messages. The Message Test Program is easily modified to adapt to different product situations, and messages can be evaluated even before working code exists. The Message Test Program can be used to test error messages for a batch product, as well as messages for an interactive product. It can also be used for stand-alone messages, for products that offer on-line help, or messages that provide additional information in a reference manual. Message testing enables us to objectively evaluate error messages and provide specific feedback about the difficulties users encounter and how error messages can be improved.