UniGS: Unified Geometry-Aware Gaussian Splatting for Multimodal Rendering

TL;DR

UniGS tackles the challenge of high-fidelity multimodal 3D reconstruction by unifying geometry-aware Gaussian Splatting into a differentiable rasterization framework. The approach jointly renders RGB, depth, normals, and semantic logits through a CUDA-accelerated pipeline, where depth is obtained via differentiable ray-ellipsoid intersection and normals are inferred from depth with analytic gradients for geometric consistency. A learnable gradient factor and differentiable pruning reduce computational and memory costs while preserving accuracy, enabling sparse yet expressive scene representations. Extensive experiments on Replica, Mip-NeRF, Tanks & Temples, and Deep Blending demonstrate state-of-the-art performance across all modalities with competitive runtimes, highlighting the practical impact for real-time multimodal 3D reconstruction and interactive visualization.

Abstract

In this paper, we propose UniGS, a unified map representation and differentiable framework for high-fidelity multimodal 3D reconstruction based on 3D Gaussian Splatting. Our framework integrates a CUDA-accelerated rasterization pipeline capable of rendering photo-realistic RGB images, geometrically accurate depth maps, consistent surface normals, and semantic logits simultaneously. We redesign the rasterization to render depth via differentiable ray-ellipsoid intersection rather than using Gaussian centers, enabling effective optimization of rotation and scale attribute through analytic depth gradients. Furthermore, we derive the analytic gradient formulation for surface normal rendering, ensuring geometric consistency among reconstructed 3D scenes. To improve computational and storage efficiency, we introduce a learnable attribute that enables differentiable pruning of Gaussians with minimal contribution during training. Quantitative and qualitative experiments demonstrate state-of-the-art reconstruction accuracy across all modalities, validating the efficacy of our geometry-aware paradigm. Source code and multimodal viewer will be available on GitHub.

Figures (9)

• Figure 1: We propose a unified 3D Gaussian Splatting framework that jointly predicts RGB, depth, normal, and semantic map through a single forward rasterization process. (a) Our geometry-aware rendering method explicitly incorporates rotation and scaling attribute into both the rasterization and gradient propagation processes, where we derive an analytical solution for gradient propagation to significantly accelerate optimization. (b) We present multimodal rendering results across diverse scenes, showcasing RGB images, depth maps, surface normals, and semantic segmentation map in a four-panel visualization layout. (c) A comprehensive comparison with widely-used baselines in Replica-Room0 dataset, underscoring the full modality coverage and strong performance of our framework. All scores are normalized to the [0, 1] range by defining the worst and best reference values for each metric individually, ensuring 1 represents optimal performance. Incomplete dashed lines indicate that the corresponding baseline does not support specific modalities. Evaluation covers PSNR for RGB quality, absolute relative error (Abs.Rel.) for depth estimation, cosine similarity (CosSimi) for normal estimation, mean intersection-over-union (mIoU) for semantic segmentation, as well as system performance metrics including the number of Gaussian primitives (Count) and rendering speed in frames per second (FPS).
• Figure 2: Overview of our framework. The yellow arrows ($\textcolor{#FFCC3D}{\rightarrow}$) indicate the forward rasterization pipeline. (a) First, we predict the semantic logits $\textbf{o}$, contribution $k$, and other 3DGS attributes (i.e., position $\boldsymbol\mu$, rotation $\textbf{q}$, scale $\textbf{s}$, opacity $\alpha$, SHs $\textbf{h}$) in the anchor set through MLPs. For each anchor, we predict $M$ raw 3DGS data available for rendering. (b) Then, during the rasterization stage, for each pixel, we use ray tracing to calculate its intersection ($\times$) with the Gaussian ellipsoid, which is closely related to the rotation and scale attributes. We replace the depth of the Gaussian center point ($\bullet$) with the depth of the midpoint ($\blacktriangle$) between the two intersection points. After splatting rasterization, we obtain RGB (①), gradient factor map (②), semantic (③), and depth (④). (c) To obtain the surface normal (⑤), we back-project the depth into three-dimensional space and then use a differential method to obtain the normals in the world coordinate system, ultimately producing the normal rendering map. The final rendered results are compared with their respective ground truths to calculate the loss. The green arrows ($\textcolor{#80CB60}{\leftarrow}$) indicate the direction of gradient propagation through a CUDA-accelerated analytical solution.
• Figure 3: Across various dataset tests, our method demonstrates impressive results in geometric reconstruction. Even in textureless areas and edge regions, it remains capable of recovering fine geometric details.
• Figure 4: A qualitative demonstration showing that the Gaussian primitives reconstructed by our method are geometrically closer to the real surface.
• Figure 5: Quantitative comparison between our method and SGS-SLAM sgsslam in terms of RGB rendering and semantic segmentation clearly shows that our approach outperforms SGS-SLAM. The top-left corner of the image displays the PSNR value computed against the ground truth image.