BT-GANformer: A generative ensemble transformer mechanism for brain tumor segmentation and classification

Abstract

The segmentation task for brain tumors from magnetic resonance imaging (MRI) has been both challenging and crucial for radiologists in their decision-making process. Recent developments in attention mechanisms for natural language processing tasks have gained wide popularity and have shown potential applications in computer vision and related problems. This article proposes a generative ensembled vision transformer that achieves state-of-the-art (SOTA) performance in segmenting brain tumors from multiple modalities of MRI scans. The proposed method includes an encoder and decoder block with CNN and transformer components, forming the generative architecture. The discriminator distinguishes the predictions of the generator from the ground truth and consists of convolutional layers along with a softmax for the classification tasks. The model was trained using the BraTS 2021 Task 1 dataset for the segmentation, and the Task 2 dataset was applied to evaluate the classification task. The proposed model scores a DICE average of 91% in tumor-core (TC), enhancing-tumor (ET), and whole-tumor (WT) categories. The model also achieves a 99% ROC AUC score in the methylguanine‐methyltransferase (MGMT) classification task.