MedTet: An Online Motion Model for 4D Heart Reconstruction

TL;DR

This work addresses the challenge of reconstructing coherent 3D cardiac motion in real time from sparse intraoperative data. It introduces MedTet, a hybrid framework that uses Deep Marching Tetrahedra (DMTet) to represent deformable tetrahedral grids with per-vertex signed distance values, enabling unlimited implicit resolution while preserving explicit motion control. A universal observation encoder processes full 3D volumes, sparse 2D slices, or 1D signals, and a GCN/GRU-based motion model propagates deformations from an initial pre-operative model, trained in a weakly supervised manner with reconstruction and motion losses. The results show MedTet can produce statistically plausible, anatomically consistent 4D reconstructions from partial observations, with near real-time performance, and offer multimodal integration potential for broader cardiac imaging and beyond. The approach balances the strengths of explicit mesh-based motion and implicit shape representations, and its applicability to other deforming organs suggests wide practical impact in intraoperative guidance and computer-assisted interventions.

Abstract

We present a novel approach to reconstruction of 3D cardiac motion from sparse intraoperative data. While existing methods can accurately reconstruct 3D organ geometries from full 3D volumetric imaging, they cannot be used during surgical interventions where usually limited observed data, such as a few 2D frames or 1D signals, is available in real-time. We propose a versatile framework for reconstructing 3D motion from such partial data. It discretizes the 3D space into a deformable tetrahedral grid with signed distance values, providing implicit unlimited resolution while maintaining explicit control over motion dynamics. Given an initial 3D model reconstructed from pre-operative full volumetric data, our system, equipped with an universal observation encoder, can reconstruct coherent 3D cardiac motion from full 3D volumes, a few 2D MRI slices or even 1D signals. Extensive experiments on cardiac intervention scenarios demonstrate our ability to generate plausible and anatomically consistent 3D motion reconstructions from various sparse real-time observations, highlighting its potential for multimodal cardiac imaging. Our code and model will be made available at https://github.com/Scalsol/MedTet.

Figures (12)

• Figure 1: Online cardiac motion recovery from full or partial observations. Given an initial 3D model represented using deep tetrahedra Shen21a, our method recovers its shape at time $t$ using either full 3D volumes, a few 2D slices, or even sparser 1D signals, such as time-volume curves.
• Figure 2: MedTet recovers cardiac motion from full or partial online observations. Initially, a 3D image volume is used to generate a tetrahedral 3D model $\mathcal{G}^0$ at $t=0$. Next, when sparse 2D slices are used, unobserved areas are zero-padded to form pseudo-volume, allowing an encoder to handle different number of slices, with full 3D being a special case. When using 1D observations, it is broadcast to each vertex in $\mathcal{G}^0$. Finally, for motion recovery at time $t$, we only predict offsets from $\mathcal{G}^0$, enabling efficient forward-flow deformation.
• Figure 3: Motion reconstruction results (3-slice) using different methods on 4DM by deforming from ED frame to the target frame. MeshDeformNet* and DeepCSR* could not faithfully reconstruct the deformation of the object, whereas our approach does.
• Figure 4: Motion reconstruction results (3-slice) using different methods on ACDC obtained by deforming from ED to ES.
• Figure 5: Accuracy as a function of the number of slices.