COVID-19 on chest radiographs: A multireader evaluation of an artificial intelligence system

Abstract

Background: Chest radiography may play an important role in triage for coronavirus disease 2019 (COVID-19), particularly in low-resource settings. Purpose: To evaluate the performance of an artificial intelligence (AI) system for detection of COVID-19 pneumonia on chest radiographs. Materials and Methods: An AI system (CAD4COVID-XRay) was trained on 24 678 chest radiographs, including 1540 used only for validation while training. The test set consisted of a set of continuously acquired chest radiographs (n = 454) obtained in patients suspected of having COVID-19 pneumonia between March 4 and April 6, 2020, at one center (223 patients with positive reverse transcription polymerase chain reaction [RT-PCR] results, 231 with negative RT-PCR results). Radiographs were independently analyzed by six readers and by the AI system. Diagnostic performance was analyzed with the receiver operating characteristic curve. Results: For the test set, the mean age of patients was 67 years 6 14.4 (standard deviation) (56% male). With RT-PCR test results as the reference standard, the AI system correctly classified chest radiographs as COVID-19 pneumonia with an area under the receiver operating characteristic curve of 0.81. The system significantly outperformed each reader (P , .001 using the McNemar test) at their highest possible sensitivities. At their lowest sensitivities, only one reader significantly outperformed the AI system (P = .04). Conclusion: The performance of an artificial intelligence system in the detection of coronavirus disease 2019 on chest radiographs was comparable with that of six independent readers.