DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos

TL;DR

DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS), is presented.

Abstract

We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. While neural rendering techniques have made significant strides in driving scenarios, existing methods are primarily designed for videos collected by autonomous vehicles. However, these videos are limited in both quantity and diversity compared to dash cam videos, which are more widely used across various types of vehicles and capture a broader range of scenarios. Dash cam videos often suffer from severe obstructions such as reflections and occlusions on the windshields, which significantly impede the application of neural rendering techniques. To address this challenge, we develop DC-Gaussian based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS). Our approach includes an adaptive image decomposition module to model reflections and occlusions in a unified manner. Additionally, we introduce illumination-aware obstruction modeling to manage reflections and occlusions under varying lighting conditions. Lastly, we employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Experiments on self-captured and public dash cam videos show that our method not only achieves state-of-the-art performance in novel view synthesis, but also accurately reconstructing captured scenes getting rid of obstructions. See the project page for code, data: https://linhanwang.github.io/dcgaussian/.

Figures (13)

• Figure 1: Given a sequence of video captured by a dash cam that may contain obstructions like reflections and occlusions, DC-Gaussian achieves high-fidelity novel view synthesis getting rid of the obstructions. (a) dash cam; (b) original video frame; (c) novel view rendering with obstruction removal.
• Figure 2: Common obstructions on windshields: (a) Mobile-phone holder; (b) Reflections; (c) Stains.
• Figure 3: Overview of DC-Gaussian framework. To model obstructions with different opacities in a unified manner, we use an learnable opacity map to adaptively reweight the contribution of transmission. The global-shared multiresolution hash encoding is introduced to fully utilize the static motion prior of obstructions. We propose a Latent Intensity Modulation module to grasp the intensity changes of reflections conditioned on camera positions. Finally, in the G3 Enhancement module, we run geometry filtering on obstruction-suppressed images to enhance the geometry of 3D Gaussians.
• Figure 4: When the intensity of incident light changes, the strength of reflections also changes accordingly (a, d). Our method achieves high-fidelity reflections synthesis (c, f) and reasonable decomposition results (b, e) under varying light. The reflections in (f) are too weak to be seen by the eye, so we brighten it to reveal the details.
• Figure 5: Comparisons with 3DGS on novel view synthesis. Because the obstructions violate multi-view consistency, the performance of 3DGS degrades significantly, resulting in artifacts and blurry renderings (highlighted by red arrows). In contrast, our method not only faithfully synthesizes novel view renderings but also renders transmission with fine details, exhibiting an improvement of 3.05dB in terms of PSNR.