Car-GS: Addressing Reflective and Transparent Surface Challenges in 3D Car Reconstruction

TL;DR

Car-GS tackles the challenge of reconstructing 3D car geometry when surfaces are highly reflective and transparent. It introduces view-dependent Gaussians to model view-specific reflections, a learnable geometry-specific opacity to decouple depth/normal rendering from RGB appearance, and a quality-aware supervision module that leverages pretrained normal priors to refine normals adaptively. The approach decouples reflectance and geometry, improving both geometry accuracy and rendering quality, and demonstrates state-of-the-art results on 3DRealCar and Ref-NeRF Real datasets with efficient training. This work has practical implications for autonomous driving simulations and VR/AR applications that require high-fidelity car reconstructions in challenging lighting conditions.

Abstract

3D car modeling is crucial for applications in autonomous driving systems, virtual and augmented reality, and gaming. However, due to the distinctive properties of cars, such as highly reflective and transparent surface materials, existing methods often struggle to achieve accurate 3D car reconstruction.To address these limitations, we propose Car-GS, a novel approach designed to mitigate the effects of specular highlights and the coupling of RGB and geometry in 3D geometric and shading reconstruction (3DGS). Our method incorporates three key innovations: First, we introduce view-dependent Gaussian primitives to effectively model surface reflections. Second, we identify the limitations of using a shared opacity parameter for both image rendering and geometric attributes when modeling transparent objects. To overcome this, we assign a learnable geometry-specific opacity to each 2D Gaussian primitive, dedicated solely to rendering depth and normals. Third, we observe that reconstruction errors are most prominent when the camera view is nearly orthogonal to glass surfaces. To address this issue, we develop a quality-aware supervision module that adaptively leverages normal priors from a pre-trained large-scale normal model.Experimental results demonstrate that Car-GS achieves precise reconstruction of car surfaces and significantly outperforms prior methods. The project page is available at https://lcc815.github.io/Car-GS.

Figures (7)

• Figure 1: Comparison of various methods based on training time and chamfer distance(CD) error. Our approach not only achieves the highest accuracy but also demonstrates a relatively short training time, highlighting its balance between superior performance and efficiency. This makes it well-suited for real-time applications.
• Figure 2: Our Car-GS method accurately separates reflections and recovers surface normals, achieves photo-realistic rendering, and demonstrates superior reconstruction quality of the car geometry, including detailed handling of reflective and transparent regions such as the car's body and windshield.
• Figure 3: Overview of Car-GS. We initialize View-Dependent Gaussians (VDG) and View-Shared Gaussians (VSG) using monocular depth estimates aligned with structure-from-motion (SfM). VDG models view-specific attributes, while VSG captures shared information. During rendering, a learnable hybrid opacity is applied to RGB images and depth/normal maps. Additionally, a quality-aware supervision module leverages pretrained normal priors to regulate rendered normals, especially in reflective regions.
• Figure 4: (a), (b), and (c) represent the input image, the normal map from the pretrained model ye2024stablenormal, and the loss mask computed by \ref{['eq:qam']}, respectively. In the mask, black indicates a value of 0, while white indicates a value of 1. We observe that errors in the pretrained model's predictions are most prominent when the camera view is nearly orthogonal to the glass surface. Therefore, we adaptively assign a value of 0 to these orthogonal regions.
• Figure 5: Visual comparisons on test-set views from the 3DRealCar dataset. Note that we focus on the reconstruction of the vehicle body rather than the entire scene. Our method excels at synthesizing geometrically accurate radiance fields and surface reconstructions, outperforming other baseline approaches in capturing sharp edges and intricate details. In contrast, baseline models often fail in areas with transparent glass and reflective surfaces on the vehicle body.