Multi-modal Broad Learning System for Medical Image and Text-based Classification

Abstract

Automatic classification of medical images plays an essential role in computer-aided diagnosis. However, the medical images arise from the small number of available data and the improvement of existing data-enhancement methods are limited. In order to fulfil this demand, a Multi-Modal Broad Learning System (M 2 -BLS) is proposed, which has two subnetworks for simultaneous learning of both medical images and the corresponding radiology reports. M 2 -BLS provides two advantages: i) our M 2 -BLS has closed-form solution and avoids iterative training, once the image feature is available; ii) benefit from the simultaneous learning of both image and text data, our M 2 -BLS achieves high accuracy for medical classification. Experimental results on the publicly available datasets IU X-RAY and PEIR GROSS-895 show that our M 2 -BLS highly improves the classification performance, compared to SOTA deep models that learn single-type of data information only.