NucDETR: End-to-End Transformer for Nucleus Detection in Histopathology Images

Abstract

Nucleus detection in histopathology images is an instrumental step for the assessment of a tumor. Nonetheless, nucleus detection is a laborious and expensive task if done manually by experienced clinicians, and is also prone to subjectivity and inconsistency. Alternatively, the advancement in computer vision-based analysis enables the automatic detection of cancerous nuclei; however, the task poses several challenges due to the heterogeneity in the morphology and color of the nuclei, their varying chromatin distribution, and their fuzzy boundaries. In this work, we propose the usage of transformer-based detection, and dub it NucDETR, to tackle this problem, given their promising results and simple architecture on several tasks including object detection. We inspire from the recently-proposed Detection Transformer (DETR), and propose the introduction of a necessary data synthesis step; demonstrating its effectiveness and benchmarking the performance of Transformer detectors on histopathology images. Where applicable, we also propose remedies that mitigate some of the issues faced when adopting such Transformer-based detection. The proposed end-to-end architecture avoids much of the post-processing steps demanded by most current detectors, and outperforms the state-of-the-art methods on two popular datasets by 1–9% in the F-score.