The Application of Simple, Practical and Affordable Standard Setting Methods in Small Groups of Students

Abstract

This article was migrated. The article was marked as recommended. BackgroundA simple method of examination standard setting, the Cohen method, and a modification of it have been previously described, in which the exam pass mark is taken as a fixed percentage of the mark obtained by near-to-top performing students. These methods have proved effective in large cohorts, but their application to smaller groups has not been documented.AimsTo retrospectively apply the two methods to the exams and tests of eight cohorts of around 50 students from years 1 and 2 of the MBBS course of a new African medical school.MethodsThe standard setting methods were applied to historical data from 127 tests and exams. Failure rates resulting from their use were compared to the failure rates resulting from the fixed pass mark of 50%.Cumulative distribution function charts (CDFs) were obtained from all tests to assess the validity of the reference points used to calculate pass marks.CDFs were used to assess the power of tests to discriminate between students. The performance of Cohen - type standardisation methods in tests of different discriminating ability is analysed and discussed.ResultsBoth methods reduced the average failure rates resulting from tests and greatly improved the consistency of failure rates. The methods do not produce valid pass marks in tests which discriminate poorly.ConclusionThe simple Cohen and modified Cohen standard setting methods can be used successfully in small groups. The methods produce fairer results than fixed pass marks or norm referenced methods but may produce anomalous results in tests which are extremely discriminating or which discriminate poorly.