Evaluating freshman engineering design projects using adaptive comparative judgment

Abstract

This evidence-based practice paper examines the use of an alternative form of assessment for engineering design projects called adaptive comparative judgment (ACJ). The authors employed ACJ to assess undergraduate engineering student design projects and compared the results to traditional marking assessment techniques. The authors sought to examine reliability and validity of ACJ in comparison with traditional assessment techniques. Student work from 16 first-year engineering majors was initially graded by the course instructor and then a panel of five judges completed the ACJ method to evaluate the same work. This work consisted of design portfolios and pictures of design prototypes. The authors conducted an analysis of the reliability and validity of the ACJ when compared to the performance data of each student's prototype and the traditional rubric used by the course instructor to evaluate the project. This paper aims to introduce the method of using ACJ for engineering design projects and make the case for this method based on current research efforts and the preliminary findings of this study.