GauSSmart: Enhanced 3D Reconstruction through 2D Foundation Models and Geometric Filtering

TL;DR

GauSSmart addresses the challenge of reconstructing detailed, realistic 3D scenes from sparsely sampled 3D data by coupling 2D foundation-model cues with 3D Gaussian Splatting. The approach integrates convex outlier filtering, segmentation-guided densification, and a DINOv3-based embedding loss to refine Gaussian primitives, enabling better semantic and geometric alignment. Experiments across DTU, Mip-NeRF 360, and Tanks & Temples show consistent improvements over 2D Gaussian Splatting and competitive results with 3D Splatting, demonstrating the value of hybrid 2D-3D priors for scene reconstruction. The work highlights a practical path toward higher fidelity 3D reconstructions in data-scarce regimes and suggests that future gains may come from leveraging newer 2D foundation models like DiNOv3.

Abstract

Scene reconstruction has emerged as a central challenge in computer vision, with approaches such as Neural Radiance Fields (NeRF) and Gaussian Splatting achieving remarkable progress. While Gaussian Splatting demonstrates strong performance on large-scale datasets, it often struggles to capture fine details or maintain realism in regions with sparse coverage, largely due to the inherent limitations of sparse 3D training data. In this work, we propose GauSSmart, a hybrid method that effectively bridges 2D foundational models and 3D Gaussian Splatting reconstruction. Our approach integrates established 2D computer vision techniques, including convex filtering and semantic feature supervision from foundational models such as DINO, to enhance Gaussian-based scene reconstruction. By leveraging 2D segmentation priors and high-dimensional feature embeddings, our method guides the densification and refinement of Gaussian splats, improving coverage in underrepresented areas and preserving intricate structural details. We validate our approach across three datasets, where GauSSmart consistently outperforms existing Gaussian Splatting in the majority of evaluated scenes. Our results demonstrate the significant potential of hybrid 2D-3D approaches, highlighting how the thoughtful combination of 2D foundational models with 3D reconstruction pipelines can overcome the limitations inherent in either approach alone.

Figures (10)

• Figure 1: Our 3D reconstruction pipeline: An input image undergoes segmentation, which enhances the 2D Gaussian splats, leading to a more geometric 3D mesh.
• Figure 2: Overview of the GauSSmart pipeline, illustrating the stages of camera clustering, Convex Hull–based outlier removal, point cloud upsampling, and the integration of a DINOv3-based loss.
• Figure 3: Visualization of a DTU scene point cloud from different viewpoints, showing point cloud outliers even on closed environments
• Figure 4: Visualization of the convex hull removal process. Left: the original point cloud containing outliers projected outside the valid viewing cone. Right: the refined point cloud after removing convex hull outliers, which eliminates spurious points and preserves only the consistent geometry.
• Figure 5: DINOv3 similarity heatmaps across DTU, NeRF, and TYT scenes, highlighting the regions most attended by the model. Brighter areas correspond to stronger alignment with the global embedding.