Identification of diagnostic discrepancies as a quality assurance measure in emergency medicine – a validation study

Abstract

Background: Diagnostic errors are a major care health concern but remain difficult to study because their identification often requires resource-intensive chart reviews. We aimed to validate a previously proposed automated method for detecting discrepancies between an initial and a later, more definitive diagnosis as a screening tool for potential diagnostic errors in a large, prospective cohort of emergency department (ED) patients. Methods: This secondary analysis included 1,204 patients enrolled in the DDxBRO randomized trial, which evaluated the effect of a diagnostic decision support tool on diagnostic quality in four Swiss emergency departments. For each patient, the ED diagnosis was extracted from the ED discharge letter, and the follow-up diagnosis at 14 days was obtained from hospital discharge letters, or general practitioner notes. All diagnoses were coded using ICD-10 and manually classified for discrepancies by two blinded ED physicians according to a predefined scheme. The automated method calculated the “similarity” between ICD-10 codes for ED and follow-up diagnoses. Discriminative performance of this method to distinguish between cases with and without diagnostic error was evaluated using receiver operating characteristic (ROC) curves, and sensitivity, specificity, and predictive values were assessed across multiple cutoffs. Results: The automated method showed high and consistent discriminative performance across all algorithms tested, with areas under the ROC curve (AUCs) ranging from 0.94 to 0.95. Using the most sensitive cutoff in the simplest algorithm, all true discrepancies were detected, but 162 cases (15%) were incorrectly flagged as discrepant. Conclusion: The automated method demonstrated high accuracy and shows promise as a practical screening tool to prioritize cases for resource-intensive chart review. Trial registration: NCT05346523.