Point2Mesh-Net: Combining Point Cloud and Mesh-Based Deep Learning for Cardiac Shape Reconstruction

Abstract

Cine magnetic resonance imaging (MRI) is the gold standard modality for the assessment of cardiac anatomy and function. However, a standard cine acquisition typically consists of only a set of intersecting 2D image slices to represent the true 3D geometry of the human heart, thus limiting its utility in various clinical and research settings. In this work, we present a novel geometric deep learning method, Point2Mesh-Net, to directly and efficiently transform a set of 2D MRI slices into 3D cardiac surface meshes. Its architecture consists of an encoder and a decoder, which are based on recent advances in point cloud and mesh-based deep learning, respectively. This allows the network to not only directly process point cloud data, which represents the sparse MRI contours obtained from image segmentation, but also to output 3D triangular surface meshes, which are highly suitable for a variety of follow-up tasks. Furthermore, the Point2Mesh-Net’s hierarchical setup with multiple downsampling and upsampling steps enables multi-scale feature learning and helps the network to successfully overcome the two main challenges of cardiac surface reconstruction: data sparsity and slice misalignment. We evaluate the model on a synthetic dataset derived from a 3D MRI-based statistical shape model and find surface distances between reconstructed and gold standard meshes below the underlying image resolution for multiple anatomical substructures of the heart. In addition, we apply the pre-trained Point2Mesh-Net as part of a multi-step pipeline to cine MRI acquisitions of the UK Biobank dataset and observe realistic mesh reconstructions with various clinical metrics in line with corresponding findings of large-scale population studies.