Joint Learning with Local and Global Consistency for Improved Medical Image Segmentation

Abstract

Semantic segmentation has been one of the key components in subsequent image-based decision-making across computer vision and biomedical imaging. While a lot of progress has been made with the advent of deep learning, segmentation models rely heavily on large labeled datasets for optimal results. Moreover, with added challenges due to varying imaging conditions, abnormalities, etc., it becomes relatively a harder problem to solve, even by the most sophisticated models. Additionally, segmentation models when employed at small patch-level lose the global context and when employed at the full image-level may lose focus to closely located and small objects-of-interest. In order to resolve such issues and thereby improve the segmentation performance, we propose a novel joint patch- and image-level training framework namely Image-to-Patch w/ Patch-to-Image (IPPI) which at the same time preserves the global context and pays attention to local details. Accommodating the joint training, our proposed IPPI technique can be incorporated with any segmentation network for improved performance and local-global consistency. Our experimentation with three different segmentation networks (U-Net, U-Net++, and NodeU-Net) in segmenting cell nuclei and retinal vessel demonstrates the effectiveness of the proposed IPPI method. The segmentation improvements—13.35% over U-Net, 5.56% over U-Net++, and 4.59% over NodeU-Net IoU (Intersection over Union) make it a potentially beneficial tool in challenging segmentation tasks.