The Development and Validation of a K-12 STEM Engagement Participant Outcome Instrument

Abstract

The U.S. Federal STEM Strategic Plan released in 2018 charged federal agencies to operate with transparency and accountability regarding the impact of STEM programming on participant outcomes. This study’s purpose is to share a robust and iterative design-based research validation study for a middle school (U.S. grades 6–8; ages 11–14 years old) Student STEM Outcomes Survey. Our team partnered with NASA to develop an instrument to study the impact of participation in NASA Office of STEM Education’s (OSTEM) engagement programming on middle school student affective outcomes. Overall, this study produced strong validity evidence for each construct (STEM Identity, STEM Self-Efficacy, STEM Interest, 21st century skills) of the Student STEM Outcomes Survey. Qualitative field testing results from subject matter experts and middle grade students related to content, response processes, and consequences of testing validity evidence provided support for data-informed item wording modifications. Rasch psychometric results assisted in meaningfully paring back items to ultimately result in parsimonious and psychometrically sound survey sections based on internal structure and response processes findings. Suggestions for using the newly developed and validated Student STEM Outcomes Survey are provided.