Development of a summative examination with subject matter expert validation

Abstract

Objective. To describe the development, implementation and impact of a summative examination on student learning and programmatic curricular outcomes. Methods. The summative examination was developed using a systematic approach. Item reliability was evaluated using standard psychometric analyses. Content validity was assessed using necessity scoring as determined by subject matter experts. Results. Almost 700 items written by 37 faculty members were evaluated. Passing standards increased annually (45% in 2009 to 67% in 2014) as the result of targeting item difficulty and necessity scores. The percentage of items exhibiting discrimination above 0.1 increased to 100% over the four years. Necessity scores above 2.75 out of 4 increased from 65% to 100% of items over six years of examination administration. Conclusion. This examination successfully assessed student and curricular outcomes. Faculty member engagement observed in this process supports a culture of assessment. This type of examination could be beneficial to other programs.