DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark

TL;DR

The paper tackles producing photorealistic scene representations under poor illumination with a moving light on a robot. It introduces NeLiS to learn a data-driven illumination model and calibrate the camera-light system, and DarkGS to build relightable 3D Gaussian scenes that render well from new viewpoints. NeLiS jointly learns the incident radiance distribution, a learnable light falloff, and ambient light, enabling accurate shading in challenging lighting. Experiments on real robots show robust reconstruction and plausible relighting, though they note limitations with shadows and non-Lambertian effects and point to future improvements.

Abstract

Humans have the remarkable ability to construct consistent mental models of an environment, even under limited or varying levels of illumination. We wish to endow robots with this same capability. In this paper, we tackle the challenge of constructing a photorealistic scene representation under poorly illuminated conditions and with a moving light source. We approach the task of modeling illumination as a learning problem, and utilize the developed illumination model to aid in scene reconstruction. We introduce an innovative framework that uses a data-driven approach, Neural Light Simulators (NeLiS), to model and calibrate the camera-light system. Furthermore, we present DarkGS, a method that applies NeLiS to create a relightable 3D Gaussian scene model capable of real-time, photorealistic rendering from novel viewpoints. We show the applicability and robustness of our proposed simulator and system in a variety of real-world environments.

Figures (10)

• Figure 1: Robotic imaging systems working in the dark consist of cameras and light sources. Examples as shown in (a): Carla Simulator carla, Team CoStar in SubT Challenge costarnebula and HoloOcean underwater robot simulator holoocean. In this work, we propose a pipeline that calibrates the camera-light system which helps photorealistic scene reconstruction and relighting from images collected in the dark.
• Figure 2: Our proposed workflow: Images for camera-light calibration are first collected at a calibration target. With NeLiS, we manually initialize the parameters and then optimize the light model. The model can then be used to build DarkGS, present the scene with learned or relighted illumination. (Learning Gamma tone mapping is supported by NeLiS but not further discussed in this paper.)
• Figure 3: Our shading model: (Left) In NeLiS, camera poses are localized by AprilTags on the calibration target. The pose of light $R_l^c$ and $t_l^c$, albedo $\mathbf{c}$ of the calibration target, ambient light $A$, RID $\Phi_\theta$, and light falloff function $\Psi_\tau$ will be learned. (Right) In building DarkGS, each Gaussian $g_i$ is modeled with an albedo $\mathbf{c}_i$ and normal $\mathbf{n}_i$. The ambient light $A$ and scale $s$ will also be optimized in this process.
• Figure 4: Our experiment setup: The imaging system is installed on a legged robot platform (Unitree GO1). We use a FLIR machine vision camera to stream the images in RAW format. The calibration target (as shown behind the dog) is a white wall with four AprilTags positioned in a rectangle.
• Figure 5: None of the existing methods can solve the problem: Results of Vanilla 3DGS kerbl3Dgaussians, RawNeRF mildenhall2022rawnerf Relightable 3DGS R3DG2023 show heavy artifacts and fail to converge. The key reason for the failures is that the existing method does not model the illumination change.