Improved Domain Generalization for Cell Detection in Histopathology Images via Test-Time Stain Augmentation

Abstract

Automated cell detection in histopathology images can provide a valuable tool for cancer diagnosis and prognosis, and cell detectors based on deep learning have achieved promising detection performance. However, the stain color variation of histopathology images acquired at different sites can deteriorate the performance of cell detection, where a cell detector trained on a source dataset may not perform well on a different target dataset. Existing methods that address this domain generalization problem perform stain normalization or augmentation during network training. However, such stain transformation performed during network training may still not be optimally representative of the test images from the target domain. Therefore, in this work, given a cell detector that may be trained with or without consideration of domain generalization, we seek to improve domain generalization for cell detection in histopathology images via test-time stain augmentation. Specifically, a histopathology image can be decomposed into the stain color matrix and stain density map, and we transform the test images by mixing their stain color with that of the source domain, so that the mixed images may better resemble the source images or their stain-transformed versions used for training. Since it is difficult to determine the optimal amount of the mixing, we choose to generate a number of transformed test images where the stain color mixing varies. The generated images are fed into the given detector, and the outputs are fused with a robust strategy that suppresses improper stain color mixing. The proposed method was validated on a publicly available dataset that comprises histopathology images acquired at different sites, and the results show that our method can effectively improve the generalization of cell detectors to new domains.