Item analysis for the written test of Taiwanese board certification examination in anaesthesiology using the Rasch model

Abstract

Background. On the written test of board certification examination for anaesthesiology, the probability of a question being answered correctly is subject to two main factors, item difficulty and examinee ability. Thus, item analysis can provide insight into the appropriateness of a particular test, given the ability of examinees. Methods. Study subjects were 36 Taiwanese examinees tested with 100 questions related to anaesthesiology. We used the Rasch model to perform item analysis of questions answered by each examinee to assess the effects of question difficulty and examinee ability using a common logit scale. Additionally, we evaluated test reliability and virtual failure rates under different criteria.Results. The mean examinee ability was higher than the mean item difficulty in this written test by 1.28 (sd=0.57) logit units, which means that the examinees, on average, were able to correctly answer 78% of items. The difficulty of items decreased from 4.25 to -2.43 on the logit scale, corresponding to the probability of having a correct answer from 5% to 98%. There were 60 items with difficulty lower than the least able examinee and seven difficult items beyond the most able one. The agreement of item difficulty between test developers and our Rasch model was poor (weighted κ=0.23).Conclusions. We demonstrated how to assess the construct validity and reliability of the written examination in order to provide useful information for future board certification examinations.The study was approved by the institutional review board with the following trial registered number: VGHIRB No. 97-08-14A. © 2010 The Author.