Validity evidence for the SUCCESS survey: Measuring non-cognitive and affective traits of engineering and computing students (Part II)

Abstract

This IUSE (Improving Undergraduate STEM Education) NSF (National Science Foundation) grantee poster describes our work deploying a national survey (the SUCCESS survey-Studying Underlying Characteristics of Computing and Engineering Student Success) to collect data on students' non-cognitive and affective (NCA) factors. This survey, which is the first of its kind to be launched on a national scale, measures 28 NCA factors that may contribute to student success including personality, grit, identity, mindset, motivation, stress, gratitude, mindfulness, and belongingness. Many engineering and computing students have strong incoming academic records and standardized test scores that indicate potential for success in their programs; nonetheless, many struggle when they reach university. Cognitive measures like SAT/ACT are weak predictors of academic success, and NCA measures may form the constellation of characteristics that offer further predictive power. In this paper, we present construct validity evidence from a confirmatory factor analysis for the SUCCESS survey using a national sample of n = 2672 students, as well as findings from our think-aloud interviews to support face validity. Through confirmatory factor analysis, we removed several items from our survey that did not load onto factors as expected thus improving the measurements and reducing survey length. In addition, the think-aloud interviews allowed us to adjust the wording of questions and to add further demographic options to the survey. Our future work includes using cluster analysis to develop non-cognitive profiles of our participants. We will also use our national dataset to develop predictive models for student success, defined in both academic (e.g., GPA, etc.) and non-academic terms.