Automated segmentation of nuclei in breast cancer histopathology images

Abstract

The process of Nuclei detection in high-grade breast cancer images is quite challenging in the case of image processing techniques due to certainheterogeneous characteristics of cancer nuclei such as enlarged and irregularlyshaped nuclei, highly coarse chromatin marginalized to the nuclei peripheryand visible nucleoli. Recent reviews state that existing techniques show appreciable segmentation accuracy on breast histopathology images whose nuclei are dispersed and regular in texture and shape; however, typical cancer nuclei are often clustered and have irregular texture and shape properties.This paper proposes a novel segmentation algorithm for detecting individual nuclei from Hematoxylin and Eosin (H&E) stained breast histopathology images. This detection framework estimates a nuclei saliency map using tensor voting followed by boundary extraction of the nuclei on the saliency map using a Loopy Back Propagation (LBP) algorithmon a Markov Random Field (MRF). The method was tested on both whole-slide images and frames of breast cancer histopathology images. Experimentalresults demonstrate high segmentation performance with efficient precision, recall and dice-coefficient rates, upon testing high-grade breast cancer images containing several thousand nuclei. In addition to the optimal performance on the highly complex images presented in this paper, this method also gave appreciable results in comparison with two recently published methods-Wienert et al. (2012) and Veta et al. (2013), which were tested using their own datasets.