Continuous improvement of a concept inventory: Using evidence centered design to refine the thermal and transport concept inventory

Abstract

Concept inventories (CIs) are increasingly being developed and used in engineering courses to assess student learning and understanding and to evaluate instructional practices. CIs differ from typical STEM assessments in that they tend to focus on a small set of key constructs and conceptual understandings within the domain-such as "the concept of force" in physics (FCI1); the area of "statics" (CATS2); or "digital logic" in computer science (DLCI3). The questions are frequently based on science and engineering education research, including research on misconceptions and common student errors. CIs often have substantial research guiding their development. Nevertheless, validating an assessment involves explicating the proposed uses and interpretation of test scores and marshaling evidence to support the acceptability and plausibility of particular claims about the meaning of those scores.4 As part of a larger research project we have developed an analytic framework to assess the validity of classroom assessments such as CIs.5 In developing this framework we have conducted validity analyses with four CIs: Conceptual Assessment Tool for Statics, Statistics Concept Inventory (SCI),6 Dynamics Concept Inventory (DCI),7 and TTCI. The results of the analyses indicated that the different inventories have varying levels of evidentiary support for claims as to the inferences one can make from scores.5.