Previous research has identified broad metric classes for human-automation performance in order to facilitate metric selection, as well as understanding and comparing research results. However, there is still a lack of a systematic method for selecting the most efficient set of metrics when designing evaluation experiments. This chapter identifies and presents a list of evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Based on the evaluation criteria, a comprehensive list of potential metric costs and benefits is generated. The evaluation criteria, along with the list of metric costs and benefits, and the existing generic metric classes provide a foundation for the development of a cost-benefit analysis approach that can be used for metric selection. © 2009 Springer-Verlag US.
CITATION STYLE
Donmez, B., Pina, P. E., & Cummings, M. L. (2009). Evaluation criteria for human-automation performance metrics. In Performance Evaluation and Benchmarking of Intelligent Systems (pp. 21–40). Springer Science and Business Media, LLC. https://doi.org/10.1007/978-1-4419-0492-8_2
Mendeley helps you to discover research relevant for your work.