Validation of a mental model elicitation instrument through deployment of control groups in an undergraduate engineering program

Abstract

Mental models of engineering systems contain information about the components, connections, inputs/outputs, and function of a system. It is difficult to study mental models because any mental model elicitation method inherently only provides a representation of the mental model and not a realization of the mental model itself. In order to measure engineering students' understanding of simple systems, a mental model instrument with scoring rubrics has been developed and deployed in undergraduate and graduate engineering classrooms in previous work. The research presented in this paper explores validation of this mental model elicitation approach through two control groups involving the collection of pre- and post-data without intervention. Results from the two control groups are discussed and indicate that without intervention, no learning effects take place. In addition, this work considers the addition of an explanatory example showing exactly how to complete the given tasks. Results concerning the inclusion of this example are inconclusive but give insight into the possible effects for future research. Lastly, unexpected component inclusions/exclusions are discussed as a final point of interest. This work serves as a first step towards validation of this new mental model elicitation method and the related scoring rubrics and is a contribution to ongoing research on mental models of engineering systems. As we continue to explore how students learn about engineering, it is important that educators and researchers have a way to reliably measure student understanding of various systems during their undergraduate and graduate degree programs.