Standard Setting: the application of the Receiver Operating Characteristic method

Abstract

Medical teachers must not only continue to accommodate and assimilate constant changes in medical knowledge, they must also assimilate new approaches to assess both stu-dents' declarative knowledge and procedural skills. Medical schools, postgraduate training programs and licensing bodies (e.g. the General Medical Council) must provide and oversee valid and reliable assessments for students' competence. 1 Assessments therefore should inform medical educators that students have a minimum acceptable level of competency which is also defensible. Over the past four decades educators have developed many standard setting methods in order to discriminate between competent and incompetent students. These methods identify different passing scores, which can be classified into two main groups, the test-centred and student approaches. 2, 3 Nedelesky, Ebel and Angoff methods, which are commonly used in medical education, are considered test-centred approaches. These traditional methods are grounded in the subjective judgments of standard setters. Sometimes standard setters use the results of classical test theory (e.g. item difficulty and item discrimination parameters) to judge the behaviour of a minimally competent (borderline) student. However, the test centred approaches are based on a mathematical consensus of standard setter's judgements rather than an analysis of the test questions. 4 A shortcoming of these methods is that if the standard setters are changed the passing mark could be changed. 5 In the student-centred approach, student abilities (abil-ity scores) are calculated to identify a minimally competent student. A common standard setting method of the student-centred approach is the "borderline method" which is usually used in OSCEs, where standard setters identify students who performed between competent and unsatisfactory levels using a global rating scale. The pass mark of the station is then the average of the checklist scores for borderline students. Most objective tests currently use the test-centred approaches with more attention increasingly being given to the student-centred approaches. 6 Other standard setting methods have been developed to differentiate students into two groups in order to identify the pass mark. For example it has been shown that there is a convergence between cluster analysis and the Angoff method for setting the pass mark. 5,7 Another technique is the "bookmark method", in which exam data are analysed using item response theory models. 8 However, a standard setting technique that is more suitable for assessors with a clinical background is the Receiver Operating Characteristic (ROC) method which uses post-examination data for setting the pass mark. Colliver and colleagues have already used the ROC method to set passing standards for a standardised-patient examination of clinical competence. 9,10 We feel that this is a useful standard setting method that should be more widely known among the medical education community and the purpose of this editorial is to describe and advocate the ROC method.