3D CVT-GAN: A 3D Convolutional Vision Transformer-GAN for PET Reconstruction

Abstract

To obtain high-quality positron emission tomography (PET) scans while reducing potential radiation hazards brought to patients, various generative adversarial network (GAN)-based methods have been developed to reconstruct high-quality standard-dose PET (SPET) images from low-dose PET (LPET) images. However, due to the intrinsic locality of convolution operator, these methods have failed to explore global contexts of the entire 3D PET image. In this paper, we propose a novel 3D convolutional vision transformer GAN framework, named 3D CVT-GAN, for SPET reconstruction using LPET images. Specifically, we innovatively design a generator with a hierarchical structure that uses multiple 3D CVT blocks as the encoder for feature extraction and also multiple 3D transposed CVT (TCVT) blocks as the decoder for SPET restoration, capturing both local spatial features and global contexts from different network layers. Different from the vanilla 2D vision transformer that uses linear embedding and projection, our 3D CVT and TCVT blocks employ 3D convolutional embedding and projection instead, allowing the model to overcome semantic ambiguity problem caused by the attention mechanism and further preserve spatial details. In addition, residual learning and a patch-based discriminator embedded with 3D CVT blocks are added inside and after the generator, facilitating the training process while mining more discriminative feature representations. Validation on the clinical PET dataset shows that our proposed 3D CVT-GAN outperforms the state-of-the-art methods qualitatively and quantitatively with minimal parameters.