Assessment of reliability, sensitivity, objectivity and validity of MCQ pharmacology exams as a potential output variable for predictive analysis

Abstract

Aim: The study aimed to assess the reliability, sensitivity, objectivity and validity of the multiple-choice question (MCQ) Pharmacology Exams conducted between 2009-2014 at the Faculty of Pharmacy, Medical University of Warsaw (MUW), and to analyse the correlations between its results and the chosen criteria of selection of candidates for Pharmacy. Methods: A 6-year retrospective surveillance analysis using the admissions data of candidates for Pharmacy in the years 2005-2010 (N=584) and multi-stage pharmacology exam results from the years 2009-2014 (N=757) conducted at the MUW. The analysis of the quality of the MCQ Pharmacology Exams conducted in the form of a 100-question test comprised a comparison of parameters of item easiness and discriminating power of particular exam tasks and mean test difficulty of the entire exam. Cranach’s α coefficient and Guttman split-half reliability coefficient was estimated to determine the reliability level of the MCQ exams. Results: Reliability of Pharmacology exams estimated with Cronbach’s α coefficient (values ranging between 0.786 and 0.884) as well as Guttman split-half reliability coefficient (between 0.807 and 0.894) showed diversity of internal consistency. The objectivity of assessment of students was analysed with the use of score thresholds, which demonstrated that the “passed/failed” cut-off point was always at the same level. The analysis of correlations between the results of MCQ Pharmacology Exams and admission exam showed no systematically occurring correlations between those variables. Conclusion: The analysis of reliability, sensitivity, objectivity and validity of the Pharmacology Exams conducted at the Faculty of Pharmacy, MUW demonstrated that, with all methodological limitations of retrospective studies, the scores of these exams might constitute a good output variable for a predictive assessment.