Bright-NeRF:Brightening Neural Radiance Field with Color Restoration from Low-light Raw Images

TL;DR

Bright-NeRF tackles the challenge of learning high-quality radiance fields from extremely low-light multi-view RAW images without supervised normal-light data or camera-specific ISP calibration. It jointly learns a color restoration field that models the sensor's adaptive response to low-photon illumination and a NeRF-based radiance field, aided by chromatic adaptation and local-smoothness losses to enable denoising and color fidelity. The approach introduces the LMRAW dataset and demonstrates state-of-the-art performance in novel-view synthesis and low-light enhancement with strong multi-view consistency across varying darkness levels. This ISP-free framework has practical impact for robust 3D scene reconstruction in real-world, poorly lit environments.

Abstract

Neural Radiance Fields (NeRFs) have demonstrated prominent performance in novel view synthesis. However, their input heavily relies on image acquisition under normal light conditions, making it challenging to learn accurate scene representation in low-light environments where images typically exhibit significant noise and severe color distortion. To address these challenges, we propose a novel approach, Bright-NeRF, which learns enhanced and high-quality radiance fields from multi-view low-light raw images in an unsupervised manner. Our method simultaneously achieves color restoration, denoising, and enhanced novel view synthesis. Specifically, we leverage a physically-inspired model of the sensor's response to illumination and introduce a chromatic adaptation loss to constrain the learning of response, enabling consistent color perception of objects regardless of lighting conditions. We further utilize the raw data's properties to expose the scene's intensity automatically. Additionally, we have collected a multi-view low-light raw image dataset to advance research in this field. Experimental results demonstrate that our proposed method significantly outperforms existing 2D and 3D approaches. Our code and dataset will be made publicly available.

Figures (11)

• Figure 1: Comparison of LLNeRF, RawNeRF, and our Bright-NeRF. LLNeRF is trained on ISP-finished sRGB images, struggling to recover accurate color in extremely dark conditions via the low bit-depth sRGB data. Utilizing high bit-depth linear raw data, RawNeRF relies on a pre-calibrated ISP for post-processing the rendered images, which involves a complex and costly calibration process. Using a generic ISP offers a compromise, but it fails to deliver high-quality results across different cameras and scenarios. In contrast, Bright-NeRF eliminates these limitations by estimating the adaptive sensor's response to low-photon lighting conditions, ensuring consistent color perception and producing vivid color.
• Figure 2: Method overview. In the training stage, we firstly sample a series of points $\mathbf{x}$ from each ray and feed them into $F_\sigma$ along with the direction $\mathbf{d}$ to obtain the view-dependent features $\mathbf{h}$ and density $\sigma$ for each point. The low-light color $\mathbf{c}_l$ is then calculated by an MLP $F_1$. Another MLP $F_2$ is designed to estimate the sensor's adaptive response to low-photon lighting conditions $\text{diag}(s_R, s_G, s_B)$. The chromatic adaptation loss $L_{ca}$ optimizes the estimations regarding the sensor's response to address color distortion in low-light conditions, while the local smooth loss $L_{smooth}$ ensures smoothness in the learned response map $S_k, k\in{\{R, G, B\}}$. The data loss optimizes the modeling for low-light images. In the inference stage, the estimated sensor's adaptive response $\text{diag}(s_R, s_G, s_B)$ is further applied to $\mathbf{c}_l$ to obtain distortion-corrected color and automatic exposure adjustment is achieved by utilizing the linear characteristics of raw data to increase the brightness. Through the joint learning of the low-light color field and the color restoration field, along with implicit noise smoothing and automatic exposure adjustment, a bright NeRF is eventually obtained.
• Figure 3: Qualitative comparison with state-of-the-art LLIE NeRF methods. Leveraging the sensor's adaptive responses in the color restoration field, the results of our Bright-NeRF exhibit more natural color and more accurate tone restoration.
• Figure 4: Qualitative comparison with state-of-the-art 2D LLIE methods. We show that our method effectively recovers detailed geometry and accurate colors while other methods often struggle with denoising and tend to produce incorrect colors. For these 2D methods, we use publicly available code and pre-trained model weights.
• Figure 5: Visualization of multi-view consistency in 2D LLIE methods and our method. We present cropped regions of enhancement results from the ABF dong2022abandoning and DNF jin2023dnf methods at different views. Both methods exhibit enhancement inconsistency such as the color inconsistency in the doll's "forehead" in the first column and "eye" in the second column. In contrast, our Bright-NeRF ensures high-quality enhancement while maintaining multi-view consistency.