Instantiation-Net: 3D Mesh Reconstruction from Single 2D Image for Right Ventricle

Abstract

3D shape instantiation for reconstructing the 3D shape of a target from limited 2D images or projections is an emerging technique for surgical navigation. It bridges the gap between the current 2D intra-operative image acquisition and 3D intra-operative navigation requirement in Minimally Invasive Surgery (MIS). Previously, a general and registration-free framework was proposed for 3D shape instantiation based on Kernel Partial Least Square Regression (KPLSR), requiring manually segmented anatomical structures as the pre-requisite. Two hyper-parameters including the Gaussian width and component number also need to be carefully adjusted. Deep Convolutional Neural Network (DCNN) based framework has also been proposed to reconstruct a 3D point cloud from single 2D image, with end-to-end and fully automatic learning. In this paper, an Instantiation-Net is proposed to reconstruct the 3D mesh of a target from its single 2D image, by using DCNN to extract features from the 2D image and Graph Convolutional Network (GCN) to reconstruct the 3D mesh, and using Fully Connected (FC) layers as the connection. Detailed validation on the Right Ventricle (RV), with a mean 3D distance error of 2.21mm on 27 patients, demonstrates the practical strength of the method and its potential clinical use.