Closing the loop on test creation

Abstract

New accreditation requirements focus on education as a "continuous improvement process." The most important part of such a process is that information gets fed back into the system to improve the quality of the output. This requirement is often interpreted to imply a course-level feedback loop that iterates on offerings of courses or entire academic years. This paper provides a smaller and more immediate feedback loop. This technique gives instructors feedback onthe quality of each question on a test or quiz, as well as a numeric score for the difficult of the question. A simple tool implementing this procedure can be used to help train instructors on which questions are difficult, as well as what types of questions are correlated with ability, and how to design a meaningful instrument of assessment. Performing this analysis at the end of a course offering could help demonstrate continuous improvement to accreditation committees. Performing this analysis immediately after the administration of a test or quiz can point out topics that the class as a whole have failed to understand, thus giving instructors more insight into student knowledge.