Camera Relocalization in Shadow-free Neural Radiance Fields

TL;DR

This work tackles camera relocalization under varying lighting and shadow conditions by introducing a two-stage pipeline that normalizes input images to a shadow-free canonical lighting before pose refinement. It leverages a hash-encoded NeRF trained on shadow-free data, and couples this with a shadow-removal preprocessor and a coarse-to-fine pose optimization enhanced by a truncated dynamic low-pass filter and numerical gradient averaging to stabilize learning. The approach yields state-of-the-art results on the SUMAD shadow-urban dataset and NeRF-OSR, demonstrating robust relocalization performance and fast training times. The method's practical impact lies in enabling reliable localization in outdoor, shadow-rich environments, with code and data to be released publicly.

Abstract

Camera relocalization is a crucial problem in computer vision and robotics. Recent advancements in neural radiance fields (NeRFs) have shown promise in synthesizing photo-realistic images. Several works have utilized NeRFs for refining camera poses, but they do not account for lighting changes that can affect scene appearance and shadow regions, causing a degraded pose optimization process. In this paper, we propose a two-staged pipeline that normalizes images with varying lighting and shadow conditions to improve camera relocalization. We implement our scene representation upon a hash-encoded NeRF which significantly boosts up the pose optimization process. To account for the noisy image gradient computing problem in grid-based NeRFs, we further propose a re-devised truncated dynamic low-pass filter (TDLF) and a numerical gradient averaging technique to smoothen the process. Experimental results on several datasets with varying lighting conditions demonstrate that our method achieves state-of-the-art results in camera relocalization under varying lighting conditions. Code and data will be made publicly available.

Figures (5)

• Figure 1: Illustration on the negative effects caused by shadows in images. (a) demonstrates the image error in the raw images (b and c) taken under different lighting conditions. Directly optimizing pose via rendering error may cause degraded solutions. Our proposed solution first normalizes the images with a shadow removal network. The shadow-free images and the image error are shown respectively in (d)-(f).
• Figure 2: Pipeline. Top: In the map construction process, we fit a hash-encoded NeRF onto a set of shadow-free images. Bottom: Once the NeRF is trained, we can recover the camera pose for any given test image $I^{(l')}$ by first processing the image with the same shadow removal network $\mathcal{N}_\text{shadow}$ as used in the training stage and refine the initial pose recursively with the NeRF network fixed.
• Figure 3: Illustration on the re-devised TDLF and the numerical gradient averaging techniques. During the pose optimization stage, we filter the extracted feature on its frequency domain by using a set of closed-form weighting parameters. When updating the estimated pose with SGD, we use numerical gradient averaging over the standard Autograd operators to ensure a smooth optimization process.
• Figure 4: We show the optimization process of our methods with different design choices. The rendered and observed images are stacked to demonstrate both rendering quality and pose accuracy.
• Figure 5: Result of our method optimizing pose on the NeRF-OSR dataset. (a) Original image from dataset at the ground truth pose. (b) Images rendered with optimized noisy pose.