A critical analysis of the body of work method for setting cut-scores

Abstract

The recent increase in the use of constructed-response items in educational assessment and the dissatisfaction with the nature of the decision that the judges must make using traditional standard-setting methods created a need to develop new and effective standard-setting procedures for tests that include both multiple-choice and constructed-response items. The Body of Work (BoW) method is an examinee-centered method for setting cut-scores that applies a holistic approach to student work in order to estimate the cut-scores that differentiate examinees according to their level of performance in situations where both item formats are used. A detailed review of Version 1 and the recent modification, Version 1, are first presented followed by a critical evaluation of the two versions in terms of Berk's (1986) 10 criteria for defensibility. The results reveal that the BoW method appears to be a promising method for setting cut-scores that could be used on a wider scale in Canada. However, as with other methods, the experience gained from using the BoW method in the field will probably lead to further modifications in an attempt to increase efficiency without sacrificing accuracy.