From 2D to 3D, Deep Learning-based Shape Reconstruction in Magnetic Resonance Imaging: A Review

TL;DR

This paper surveys deep learning methods for reconstructing 3D anatomy from 2D MRI slices, formalizing the objective as recovering a 3D shape $\hat{X}$ that approximates the ground-truth $X$ from input sets $\mathcal{I}$. It categorizes approaches into point-cloud, mesh-based, shape-aware, and volumetric models, and analyzes CNN-, GCN-, and diffusion-based architectures across organs such as the heart, brain, and lungs. Key insights include the rising prominence of diffusion models for high-fidelity reconstructions, the value of anatomical priors and template-based constraints, and ongoing challenges in data scarcity, cross-site generalization, and computational demands. The review synthesizes public datasets, evaluation metrics, and emerging directions (multimodal integration, cross-modality frameworks) to guide development of robust, generalizable, clinically impactful 3D MRI reconstruction systems.

Abstract

Deep learning-based 3-dimensional (3D) shape reconstruction from 2-dimensional (2D) magnetic resonance imaging (MRI) has become increasingly important in medical disease diagnosis, treatment planning, and computational modeling. This review surveys the methodological landscape of 3D MRI reconstruction, focusing on 4 primary approaches: point cloud, mesh-based, shape-aware, and volumetric models. For each category, we analyze the current state-of-the-art techniques, their methodological foundation, limitations, and applications across anatomical structures. We provide an extensive overview ranging from cardiac to neurological to lung imaging. We also focus on the clinical applicability of models to diseased anatomy, and the influence of their training and testing data. We examine publicly available datasets, computational demands, and evaluation metrics. Finally, we highlight the emerging research directions including multimodal integration and cross-modality frameworks. This review aims to provide researchers with a structured overview of current 3D reconstruction methodologies to identify opportunities for advancing deep learning towards more robust, generalizable, and clinically impactful solutions.

Figures (8)

• Figure 1: Example pipeline of 2D MRI to 3D reconstruction illustrating the process of image segmentation using U-Net optimizedxrayGopatoti2022, then generating sparse contours as input into a diffusion model rombach2022highresolutionimagesynthesislatent which can create a 3D mesh reconstruction of the heart qiao2025personalized. All images used under Open Access licenses from their respective publications.
• Figure 2: Example 3D reconstruction deep learning architectures. These models cannot be used for 3D reconstruction on their own, but can be used as part of a reconstruction pipeline. (a) Standard U-Net archie2021unet. (b) General Adversarial Network (GAN) Alrashedys22114297 is an end-to-end pipeline that trains the generator in an adversarial manner to generate samples that the discriminator is capable of distinguishing from the real data sample. This example is GAN-LSTM wherein after extracting features by pretrained model, a 3D model of the tumour is reconstructed hong_gan-lstm-3d_nodate (c) Diffusion models rombach2022highresolutionimagesynthesislatent define a Markov chain of diffusion steps to slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. All images used under Open Access licenses from their respective publications.
• Figure 3: Diagrams for 3D mesh generation. a) Schematic diagram of a generative adversarial network (GAN)-based model for mesh generation. 2D images are inputted into the encoder which produces a latent vector. The latent vector is then passed to the generator which generates a point cloud. The point cloud is inputted into a discriminator network which differentiates real from fake shapes. b) Schematic diagram of a diffusion model for 3D mesh generation. Diffusion models gradually transform noise into structure. A feature encoder processes the images to produce a latent vector or embedding that summarizes the organ’s 3D shape. c) Schematic diagram of a graph convolutional network (GCN). The GCN deforms a template mesh using encoded features and local graph operations. Mesh losses enforce smoothness, realism, and anatomical correctness.
• Figure 4: Examples of 3D MRI point cloud reconstructions. a) Dense point cloud and output mesh created by PCCN of the human heart ventricles beetz_multi-class_2023. b) Different angle MRI scans and the generated point clouds from SRThu2022. All images used under Open Access licenses from their respective publications.
• Figure 5: Exemplar of deep learning mesh-based 3D reconstructions. a) HybridVNet HybridVNetGaggion_2025. This image is used under Open Access licenses from their respective publications.