Evolving High-Quality Rendering and Reconstruction in a Unified Framework with Contribution-Adaptive Regularization

TL;DR

This work introduces CarGS, a unified 3D Gaussian Splatting framework that simultaneously delivers high-fidelity rendering and accurate surface reconstruction. It realizes Contribution-adaptive regularization by learning per-Gaussian contributions through a lightweight MLP, and further stabilizes learning with Lite-Geo, a geometry-aware residual for depth cues. A geometry-guided densification strategy leverages normals and SDF to capture fine details, while a joint optimization with plane and cross-view losses enforces geometric consistency. Experiments on Tanks and Temples and Mip-NeRF360 show CarGS achieving state-of-the-art performance with real-time rendering and significantly reduced storage compared to dual-model baselines, establishing a strong baseline for unified rendering and reconstruction.

Abstract

Representing 3D scenes from multiview images is a core challenge in computer vision and graphics, which requires both precise rendering and accurate reconstruction. Recently, 3D Gaussian Splatting (3DGS) has garnered significant attention for its high-quality rendering and fast inference speed. Yet, due to the unstructured and irregular nature of Gaussian point clouds, ensuring accurate geometry reconstruction remains difficult. Existing methods primarily focus on geometry regularization, with common approaches including primitive-based and dual-model frameworks. However, the former suffers from inherent conflicts between rendering and reconstruction, while the latter is computationally and storage-intensive. To address these challenges, we propose CarGS, a unified model leveraging Contribution-adaptive regularization to achieve simultaneous, high-quality rendering and surface reconstruction. The essence of our framework is learning adaptive contribution for Gaussian primitives by squeezing the knowledge from geometry regularization into a compact MLP. Additionally, we introduce a geometry-guided densification strategy with clues from both normals and Signed Distance Fields (SDF) to improve the capability of capturing high-frequency details. Our design improves the mutual learning of the two tasks, meanwhile its unified structure does not require separate models as in dual-model based approaches, guaranteeing efficiency. Extensive experiments demonstrate the ability to achieve state-of-the-art (SOTA) results in both rendering fidelity and reconstruction accuracy while maintaining real-time speed and minimal storage size.

Figures (7)

• Figure 1: The performance of CarGS in TnT datasets, achieving high-quality rendering and reconstruction in a unified model. The storage size of the model is represented by the radius of the circle.
• Figure 2: The performance of the Baseline model chen2024pgsr across various CD% thresholds demonstrates that each Gaussian primitive contributes differently to specific tasks.
• Figure 3: The illustration of contribution conflict between rendering and reconstruction tasks.
• Figure 4: The illustration of the geometry regularization GS framework: (a) Primitive based regularization paradigm. (b) Dual-model-based regularization paradigm. (c) the proposed contribution adaptative regularization paradigm.
• Figure 5: The illustration of geometry overfitting. If the Geo-MLP is solely associated with geometry reconstruction, notable overfitting is observed. By leveraging the approximate depth knowledge provided by rendering, Lite-Geo can effectively reduce overfitting and suppress noise where the Chamfer Distance exceeds the threshold.