Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images

TL;DR

This paper tackles the challenge of recovering accurate geometry and HDR radiance from sparse Low Dynamic Range panoramic inputs. It introduces irradiance fields to capture inter-reflection in panoramic scenes, providing additional constraints that enlarge effective observations for geometry and enable HDR reconstruction via irradiance-radiance attenuation. The irradiance fields are integrated with a NeRF-based radiance field (Pano-NeRF) by extending the backbone to predict an albedo-driven irradiance channel and incorporating a geometry prior, resulting in state-of-the-art geometry recovery and HDR synthesis on sparse panoramas. A notable byproduct is the ability to estimate spatially varying lighting, which improves realism for inserted objects and illuminates practical XR scenarios.

Abstract

Panoramic imaging research on geometry recovery and High Dynamic Range (HDR) reconstruction becomes a trend with the development of Extended Reality (XR). Neural Radiance Fields (NeRF) provide a promising scene representation for both tasks without requiring extensive prior data. However, in the case of inputting sparse Low Dynamic Range (LDR) panoramic images, NeRF often degrades with under-constrained geometry and is unable to reconstruct HDR radiance from LDR inputs. We observe that the radiance from each pixel in panoramic images can be modeled as both a signal to convey scene lighting information and a light source to illuminate other pixels. Hence, we propose the irradiance fields from sparse LDR panoramic images, which increases the observation counts for faithful geometry recovery and leverages the irradiance-radiance attenuation for HDR reconstruction. Extensive experiments demonstrate that the irradiance fields outperform state-of-the-art methods on both geometry recovery and HDR reconstruction and validate their effectiveness. Furthermore, we show a promising byproduct of spatially-varying lighting estimation. The code is available at https://github.com/Lu-Zhan/Pano-NeRF.

Figures (7)

• Figure 1: Comparison between the existing radiance fields (top) and the proposed irradiance fields (bottom) on geometry recovery (left) and HDR reconstruction (right) from sparse LDR panoramic images. Radiance fields suffer from poor geometry recovery due to a few observation counts (number of blue dots) and cannot reconstruct HDR radiance with LDR inputs. In contrast, the proposed irradiance fields recover faithful geometry by increasing the observation counts (number of blue dots) from incident rays (yellow lines) and infer over-saturated radiance from the unsaturated area by considering irradiance-radiance attenuation.
• Figure 2: An illustration of different outgoing radiance modeling between the radiance fields and the proposed irradiance fields. (Left) Radiance fields integrate the radiance $\mathbf{c}^r$ of each volumetric particle (blue dots) with weight $w^r$ along the camera ray $\mathbf{r}$ (original line). (Right) Irradiance fields integrate the radiance $\mathbf{c}^i$ from the incident light directions (yellow lines) with weight $f_r(\mathbf{x}, \boldsymbol{\omega}_o, \boldsymbol{\omega}_i) \cdot \boldsymbol{\omega}_i \mathbf{n}^\top$.
• Figure 3: Illustration of calculating the outgoing radiance $\mathbf{C}^{i}$ from the conventional radiance fields. Firstly, we obtain the intrinsic factors on a surface point by integrating the BRDF output $\Phi$, derived density $d\sigma$, and distance $t$ along the camera ray $\mathbf{r}$ via volume rendering, as same as the computation of radiance $\mathbf{C}^{r}$. Secondly, We sample the incident light rays $\mathbf{r}^\prime$ on the surface point $\mathbf{x}$ from directions distributed uniformly on the sphere centered at $\mathbf{x}$, and then compute the radiance $\mathbf{c}^{i}$ of sampled incident light rays. At last, we integrate obtained BRDF $f_r(\mathbf{x})$, surface normal $\mathbf{n}(\mathbf{x})$, and incident light rays (with $\mathbf{c}^{i}$ and direction $\omega^{i}$) to calculate outgoing radiance $\mathbf{C}^{i}$.
• Figure 4: Comparison of geometry recovery in term of depth maps (1st&4th rows), surface normal maps (2nd&5th rows), and panoramic images (3rd&6th rows). The ground truth of depth and surface normal for the real scene is only for visual reference.
• Figure 5: Comparison of HDR reconstruction on input views. The first column is the input LDR panoramic image, and 2nd to 5th columns show the error maps between the reconstructed HDR panoramic images and the ground truth.