Liver Tumor Segmentation with Deep Learning: A Comparative Analysis of CNN-, Transformer-, and YOLO-Based Models on the ATLAS MRI

Abstract

Background/Objectives: Hepatocellular carcinoma (HCC) remains a leading cause of cancer-related mortality worldwide, where accurate liver and tumor segmentation from magnetic resonance imaging (MRI) is essential for diagnosis, treatment planning, and disease monitoring. Despite recent advances, MRI-based segmentation remains challenging due to data heterogeneity and limited annotated datasets. This study aims to systematically compare convolutional, transformer-based, and detection-based deep learning approaches for liver and HCC segmentation using contrast-enhanced MRI. Methods: A comprehensive evaluation was conducted on the ATLAS MRI dataset, including 2D- and 3D-CNN, transformer-based architectures, and single-stage YOLO-based segmentation frameworks. All models were trained using consistent preprocessing, patient-level data splits, and standardized evaluation metrics, including Dice coefficient, Intersection over Union (IoU), precision, recall, and F1-score. Results: Volumetric convolutional models achieved the highest segmentation accuracy, with the 3D nnU-Net yielding superior performance for both liver (Dice: 0.946) and tumor (Dice: 0.892) segmentation. Transformer-based models demonstrated competitive results, particularly in capturing global contextual information and improving boundary delineation, while YOLO-based approaches provided balanced accuracy with substantially reduced computational cost. Conclusions: The findings confirm that volumetric CNNs remain the most accurate solution for MRI-based liver and HCC segmentation, whereas transformer- and YOLO-based frameworks offer complementary advantages for specific clinical and resource-constrained scenarios.