Function in engineering: Benchmarking representations and models

Abstract

This paper presents the requirements and needs for establishing a benchmarking protocol that considers representation characteristics, supported cognitive criteria, and enabled reasoning activities for the systematic comparison of function modeling representations. Problem types are defined as reverse engineering, familiar products, novel products, and single-component systems. As different modeling approaches share elements, a comparison of modeling approaches on multiple levels was also undertaken. It is recommended that researchers and developers of function modeling representations collaborate to define a canonically acceptable set of benchmark tests and evaluations so that clear benefits and weaknesses for the disparate collection of approaches can be compared. This paper is written as a call to action for the research community to begin establishing a benchmarking standard protocol for function modeling comparison purposes. This protocol should be refined with input from developers of the competing approaches in an academically open environment. At the same time, the benchmarking criteria identified should also serve as a guide for validating a modeling approach or analyzing its failure.