Robust cell detection and segmentation in histopathological images using sparse reconstruction and stacked denoising autoencoders

Abstract

Computer-aided diagnosis (CAD) is a promising tool for accurate and consistent diagnosis and prognosis. Cell detection and segmentation are essential steps for CAD. These tasks are challenging due to variations in cell shapes, touching cells, and cluttered background. In this paper, we present a cell detection and segmentation algorithm using the sparse reconstruction with trivial templates and a stacked denoising autoencoder (sDAE) trained with structured labels and discriminative losses. The sparse reconstruction handles the shape variations by representing a testing patch as a linear combination of bases in the learned dictionary. Trivial templates are used to model the touching parts. The sDAE, trained on the original data with their structured labels and discriminative losses, is used for cell segmentation. To the best of our knowledge, this is the first study to apply sparse reconstruction and sDAE with both structured labels and discriminative losses to cell detection and segmentation. It is observed that structured learning can effectively handle weak or misleading edges, and discriminative training encourages the model to learn groups of filters that activate simultaneously for different input images to ensure better segmentation. The proposed method is extensively tested on four data sets containing more than 6000 cells obtained from brain tumor, lung cancer, and breast cancer and neuroendocrine tumor (NET) images. Our algorithm achieves the best performance compared with other state of the arts.