Comments on 'MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge'

Abstract

The MoNuSAC 2020 challenge was hosted at the ISBI 2020 conference, where the winners were announced. Challenge organizers, in addition to the leaderboard, released the evaluation code and visualisations of the prediction masks of the 'top 5' teams. This shows a very high level of transparency, and provides a unique opportunity to better understand the challenge results. Our analysis of the code and all released data, however, shows three different problems in the computation of the metric used for the official ranking: a coding mistake resulting in erroneous false positives; another resulting in missed false positives; and a problem with the metric's aggregation method. We demonstrate the errors, and confirm that the mistaken version of the code was indeed used to rank the algorithms in the challenge. Our results can be fully replicated with the code provided on GitHub.