Development and Test-Item Analysis of a Freely Available 1900-Item Question Bank for Rheumatology Trainees

Abstract

Background Tests composed of multiple-choice questions are an established tool to help evaluate knowledge of medical content. Within the field of rheumatology, there is an absence of free and easily-accessible sets of multiple-choice questions that have been rigorously evaluated and analyzed. Objective To develop a question bank composed of multiple-choice questions that evaluate trainee knowledge of rheumatology, as well as to investigate the psychometric properties (reliability, discrimination indices, difficulty indices) of items within the question bank. Methods Multiple-choice questions were drafted according to a strict methodology devised by the investigators. Between January and December 2020, questions were administered in sets of 20-25 questions to test-takers who were either current trainees or had recently graduated from training programs. Performance was evaluated through descriptive statistics (mean, median, range, standard deviation) and test-item statistics (difficulty index, discrimination index, reliability). Results Investigators drafted 1900 multiple choice questions within 45 sections each composed of 20 to 25 questions each. These questions were administered to 32 participants. The mean discrimination index was 0.57 (standard deviation: 0.22) and mean difficulty index was 0.38 (standard deviation: 0.23). Reliability indices for the 45 sections ranged from 0.45 to 0.85 (mean: 0.613, standard deviation: 0.09). The overall reliability index for the entire item bank was greater than 0.95. Conclusion The investigators developed a 1900-item question bank composed of items that have sufficient difficulty and discrimination indices to be used for low- and moderate-stakes settings. A rigorous methodology was employed to create the first freely-accessible reliable tool for the assessment of rheumatology knowledge. This tool can be purposed for both summative and formative evaluation in multiple settings and platforms.