LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes

TL;DR

A novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images via multi-view feature consistency and frequency information of NeRF, respectively and outperforms existing approaches.

Abstract

Neural Radiance Fields (NeRFs) have shown remarkable performances in producing novel-view images from high-quality scene images. However, hand-held low-light photography challenges NeRFs as the captured images may simultaneously suffer from low visibility, noise, and camera shakes. While existing NeRF methods may handle either low light or motion, directly combining them or incorporating additional image-based enhancement methods does not work as these degradation factors are highly coupled. We observe that noise in low-light images is always sharp regardless of camera shakes, which implies an implicit order of these degradation factors within the image formation process. To this end, we propose in this paper a novel model, named LuSh-NeRF, which can reconstruct a clean and sharp NeRF from a group of hand-held low-light images. The key idea of LuSh-NeRF is to sequentially model noise and blur in the images via multi-view feature consistency and frequency information of NeRF, respectively. Specifically, LuSh-NeRF includes a novel Scene-Noise Decomposition (SND) module for decoupling the noise from the scene representation and a novel Camera Trajectory Prediction (CTP) module for the estimation of camera motions based on low-frequency scene information. To facilitate training and evaluations, we construct a new dataset containing both synthetic and real images. Experiments show that LuSh-NeRF outperforms existing approaches. Our code and dataset can be found here: https://github.com/quzefan/LuSh-NeRF.

Figures (9)

• Figure 1: Given a hand-held captured low-light scene (a), while (a combination of) existing low-light enhancement/NeRF methods may not produce visually pleasing novel-view images ((b)-(e)), our LuSh-NeRF can produce bright and sharp results (f).
• Figure 2: The pipeline of our proposed LuSh-NeRF. It contains two novel modules: (a) SND module: Decompose the noise in each view from the origin training image with a Noise NeRF architecture, and utilize the multi-view consistency characteristic in 3D scenario to separate the scene information and noise better; (b) CTP module: To minimize the interference of noise in low-light images on blur kernel predictions, the high frequency domain of the low light regions which are severely affected by noise are abandoned. In the rendering stage, we discard the Noise Estimator and Blur Kernel, and only use the Scenario-NeRF to render the enhanced scene.
• Figure 3: Different degradations in the real low light images. (a) Low intensity (b) Noise (c) Blur.
• Figure 4: Qualitative results of different methods on our real scenes. Our LuSh-NeRF can render cleaner and sharper results for real low-light scenes with camera motions.
• Figure 5: Qualitative results of different methods on our synthetic scenes. Our method yields the most natural restoration results while sharpening the image.