ACAT assessment of grade-based and outcome-based criteria

Abstract

Building on substantial work done on a new automated course and program assessment tool, we undertook an experiment to determine the accuracy of the outcomes assessments. We compared assessments of course-level outcomes using manual instructor assessments in Moodle and our Automated Course Assessment Tool (ACAT). Four courses taught in the Spring of 2015 were considered. Two are Laboratory courses, and two are traditional book-based courses with exams or projects. We found that in all but one specific case the manual assessment matched the ACAT assessment. In the sole case that did not match, a single indicator attempted to measure all course outcomes simultaneously. The study considered 34 students, 27 outcomes, and used 107 indicators to measure the outcomes. A single indicator affecting two students caused three outcomes to be adjudged incorrectly.