GastroNet-5M: A Multicenter Dataset for Developing Foundation Models in Gastrointestinal Endoscopy

Abstract

Background & Aims: Training deep learning systems in endoscopy generally requires vast datasets of annotated images, which are often scarce and costly to obtain. Foundation models are pretrained on large, diverse datasets and can be applied across a wide range of tasks with minimal additional fine-tuning. For endoscopy, foundation models require datasets of general endoscopic images. Yet, datasets for developing such models remain limited. In this study, we present GastroNet-5M, a dataset comprising 4,820,653 endoscopic images of ∼500,000 procedures. Methods: GastroNet-5M consists of anonymized general endoscopic images captured in 8 Dutch hospitals between 2012 and 2020. Using self-supervised learning, GastroNet-5M was used to develop a foundation model for subsequent downstream endoscopic artificial intelligence (AI) applications. We compared our GastroNet-5M foundation model with publicly available endoscopic foundation models and state-of-the-art nonfoundation models across 17 endoscopic AI applications throughout the gastrointestinal tract. Outcome measures were classification and segmentation accuracy, data efficiency, and robustness to data heterogeneity. Results: GastroNet-5M–pretrained models outperformed all other models in accuracy for nearly all classification and segmentation tasks. Furthermore, GastroNet-5M–pretrained models required significantly less application-specific training data for satisfactory model performance and displayed more robust performance when models were exposed to data heterogeneity such as imagery from different endoscope manufacturers. Conclusions: This study presents GastroNet-5M, a dataset of ∼5 million endoscopic images. Pretraining endoscopic deep learning systems with GastroNet-5M improves diagnostic accuracy, reduces the need for scarce application-specific endoscopic imagery and annotations, and increases their robustness to the inevitable data heterogeneity in clinical practice. This may significantly accelerate development and implementation of endoscopic AI systems. GastroNet-5M is publicly available for scientific use.