MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging

Abstract

Endoluminal endoscopic procedures are essential for diagnosing colorectal cancer and other severe conditions in the digestive tract, urogenital system, and airways. 3D reconstruction and novel-view synthesis from endoscopic images are promising tools for enhancing diagnosis. Moreover, integrating physiological deformations and interaction with the endoscope enables the development of simulation tools from real video data. However, constrained camera trajectories and view-dependent lighting create artifacts, leading to inaccurate or overfitted reconstructions. We present MedGS, a novel 3D reconstruction framework leveraging the unique property of endoscopic imaging, where a single light source is closely aligned with the camera. Our method separates light effects from tissue properties. MedGS enhances 3D Gaussian Splatting with a physically based relightable model. We boost the traditional light transport formulation with a specialized MLP capturing complex light-related effects while ensuring reduced artifacts and better generalization across novel views. MedGS achieves superior reconstruction quality compared to baseline methods on both public and in-house datasets. Unlike existing approaches, MedGS enables tissue modifications while preserving a physically accurate response to light, making it closer to real-world clinical use. Repository: https://github.com/gmum/MedGS

Figures (4)

• Figure 1: Our MedGS model shares a unified geometry to simultaneously solve frame interpolation and mask reconstruction tasks. This multi-task coupling leverages the visual signal from the image branch to regularize the underlying Folded-Gaussian distribution. This synergy ensures geometric continuity and enables high-fidelity 3D mesh extraction even from sparse input slices.
• Figure 2: Qualitative mesh reconstruction results on prostate ultrasound data from zachary_m_c_baum_2023_7870105, and on heart, lung, and kidney from d2024totalsegmentator. Our method demonstrates superior topology preservation, edge continuity, and overall surface geometry. FUNSR and Poisson produce less smooth meshes and fail under sparse sampling conditions.
• Figure 3: Qualitative results of frame interpolation on MRI data. MedGS produces sharper reconstructions. For the ankle MRI, the MSE is also visualized using a heat map.
• Figure 4: CTA-based 3D reconstructions for advanced disease monitoring. Left: Longitudinal tracking of an abdominal aortic aneurysm. Mass in 1st and 2nd data point is colored in light green and light orange, respectively. Right: Precise localization of a renal tumor within the kidney's cortex and medulla.