A systematic approach to evaluating design prompts in supporting experimental design research

Abstract

Experiments that study engineering behavior in design often rely on participants responding to a given design prompt or a problem statement. Moreover, researchers often find themselves testing multiple variables with a relatively small participant pool. In such situations multiple design prompts may be used to boost replication by giving each participant an equivalent problem with a different experimental condition. This paper presents a systematic approach to compare given design prompts using a two-step process that allows an initial comparison of the prompts and a post-experiment verification of the similarity of the given prompts. Comparison metrics are provided which can be used to evaluate a level of similarity of existing prompts as well as develop similar problems. These metrics include complexity (size, coupling, and solvability), familiarity, and prompt structure. Statistical methods are discussed for post-experiment verification. Guidelines are provided for a post-experiment survey which may be used for an additional perspective of prompt similarity. The proposed approach is demonstrated using an experiment where two design prompts were used for within-subject replication.