CORAL8: Concurrent object regression for area localization in medical image panels

Abstract

This work tackles the problem of generating a medical report for multi-image panels. We apply our solution to the Renal Direct Immunofluorescence (RDIF) assay which requires a pathologist to generate a report based on observations across eight different whole slide images (WSI) in concert with existing clinical features. To this end, we propose a novel attention-based multi-modal generative recurrent neural network (RNN) architecture capable of dynamically sampling image data concurrently across the RDIF panel. The proposed methodology incorporates text from the clinical notes of the requesting physician to regulate the output of the network to align with the overall clinical context. In addition, we found the importance of regularizing attention weights for the word generation processes. This is because the system can ignore the attention mechanism by assigning equal weights for all members. Thus, we propose two regularizations to encourage efficient use of the attention mechanism. Experiments on our novel collection of RDIF WSIs provided by Sullivan Nicolaides Pathology demonstrate that our framework offers significant improvements over existing methods.