Reliability and validity of short answer questions in Pharmacology

Abstract

Background: Short answer question (SAQ) is an excellent tool for assessment. Objective of this study was to understand the practices adopted in answering SAQ, assessment of reliability and validity of SAQ and faculty feedback on valuing SAQs.Methods: A cross-sectional study was done in the Department of Pharmacology of a Government Medical College in Central Kerala from December 2018-January 2019. The study participants were given 10 minutes to answer the two chosen SAQs after which the response sheets were collected and valued by 2 independent examiners. Feedback on correction of the same questions was sought from Pharmacology faculties of various Medical Colleges through online survey. The data were analysed using SPSS software.Results: 100% participants attempted to answer the SAQ ‘Definition of a drug’ but none of them came out with the World Health Organisation definition of drug. For the second SAQ, ‘Two uses of atropine’ around 25(20.8%) participants had written two uses of the drug correctly. The reliability was 0 for ‘definition of a drug’ and 0. 50 (95% CI 0.082-0.702) for ‘Two uses of atropine’ and both the questions had no validity. The feedback on valuing definition and two use of atropine was variable.Conclusions: None of the students wrote definition of a drug correctly and majority gave multiple responses to uses of atropine instead of two valid responses. The short answer questions we tested were found to have moderate to no reliability and no validity. The faculties had variable perceptions in valuing the two short answer questions.