Evaluation of deep learning for COVID-19 diagnosis: Impact of image dataset organization

Abstract

Introduction: Coronavirus disease 2019 (COVID-19) has spread all over the world showing high transmissibility. Many studies have proposed diverse diagnostic methods based on deep learning using chest X-ray images focusing on performance improvement. In reviewing them, this study noticed that evaluation results might be influenced by dataset organization. Therefore, this study identified whether the high-performance values can prove the clinical application potential. Methods: This study selected chest X-ray image databases which have been widely applied in previous studies. One database includes images for COVID-19, while the others consist of normal and pneumonia images. Then, the COVID-19 classification model was designed and trained on diverse database compositions and evaluated using confusion matrix-based metrics. Also, each database was analyzed by graphical representation methods. Results: The performance was significantly different according to dataset composition. Overall, higher performance was identified on the dataset organized with different databases for each class, compared with the dataset from same database. Also, there were significant differences in the image characteristics between different databases. Conclusions: The experimental results indicate that model may be trained based on differences of the image characteristics between databases and not on lesion features. This shows that evaluation metrics can be influenced by dataset organization, and high metric values would not directly mean the potential for clinical application. These emphasize the importance of suitable dataset organization for applying COVID-19 diagnosis methods to real clinical sites. Radiologists should sufficiently understand about this issue as actual user of these methods.