Reproducibility of clinical performance assessment in practice using incognito standardized patients

Abstract

Background: The reproducibility of authentic assessment methods has been investigated for objective structured clinical examinations (OSCEs) and video assessment in general practice, but not for assessment with incognito standardized patients. Purpose: To investigate the reproducibility of assessment with incognito standardized patients. Methods: A total of 27 Dutch rheumatologists in 16 hospitals were each visited by 8 incognito standardized patients presenting with different rheumatological disorders. After each visit, the standardized patient completed a case-specific checklist containing items on medical history, physical examination and management. Over a 20-month period, 254 incognito visits took place, of which 201 were first visits. The standardized patient was detected by the rheumatologist in 2 cases only. These encounters were not included in the analysis. Generalizability theory was used to investigate the reproducibility of the assessment. Results: One fifth of the variance can be attributed to variation between rheumatologists. The largest variance is due to the variation in difficulty among cases. A reproducible assessment requires 3 hours of testing time (6 cases) if it is obtained through a norm-referenced interpretation of scores and 7 hours of testing time (14 cases) if it is obtained through an absolute interpretation of scores. Conclusion: The reproducibility of performance assessment in clinical practice by incognito standardized patients is similar to that of other authentic measurements for the assessment of clinical competence and performance.