The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility

TL;DR

This work tackles outdoor scene inverse rendering by jointly estimating geometry, albedo, distant illumination, and sky visibility from unconstrained image collections. It introduces NeuSky, which combines a sky-pixel constrained illumination prior (RENI++), an outside-in differentiable sky visibility model based on a spherical directional distance field, and end-to-end training to allow shadow information to influence geometry and illumination estimation. The method uses a NeSDF-based geometry with two spherical neural fields and a FiLM-conditioned DDF for visibility, enabling high-fidelity relighting and accurate decomposition even when sky observations are partial. Empirical results on the NeRF-OSR relighting benchmark show state-of-the-art performance, with notable improvements in albedo/shadow separation and geometry quality, while also enabling shadows to constrain scene illumination and structure. This approach advances practical outdoor relighting and environment capture by better leveraging sky cues and differentiable visibility in a unified framework, albeit with substantial training memory and time requirements.

Abstract

Inverse rendering of outdoor scenes from unconstrained image collections is a challenging task, particularly illumination/albedo ambiguities and occlusion of the illumination environment (shadowing) caused by geometry. However, there are many cues in an image that can aid in the disentanglement of geometry, albedo and shadows. Whilst sky is frequently masked out in state-of-the-art methods, we exploit the fact that any sky pixel provides a direct observation of distant lighting in the corresponding direction and, via a neural illumination prior, a statistical cue to derive the remaining illumination environment. The incorporation of our illumination prior is enabled by a novel `outside-in' method for computing differentiable sky visibility based on a neural directional distance function. This is highly efficient and can be trained in parallel with the neural scene representation, allowing gradients from appearance loss to flow from shadows to influence the estimation of illumination and geometry. Our method estimates high-quality albedo, geometry, illumination and sky visibility, achieving state-of-the-art results on the NeRF-OSR relighting benchmark. Our code and models can be found at https://github.com/JADGardner/neusky

Figures (14)

• Figure 1: From in-the-wild, outdoor image collections, we predict scene geometry, albedo, distant environment illumination, and sky visibility. Sky visibility and illumination are both modelled via spherical neural fields whereby we directly constrain illumination via sky pixel observations. Our outside-in differentiable visibility enables estimation of cast shadows and avoids shadow baking into albedo.
• Figure 2: We surround our NeuS-Facto yu_sdfstudio_2022 volume with two spherical neural fields at radius $1$ and radius $\infty$ modelling sky visibility and distant illumination respectively. Blue arrows correspond to rays sampling distant illumination. Pink circles and Maroon arrows are position and direction samples of our sky visibility network. In a given direction, visibility changes with position but distant illumination does not. For speed we only sample sky visibility on the surface of our scene, Green circles, and distribute this visibility to all samples, Orange circles, along a ray.
• Figure 3: We model our illumination and illumination visibility via two spherical neural fields at radius $\infty$ and $1$ respectively. However our world space is contracted as per Mip-NeRF-360 barron_mip-nerf_2022, such that any point at infinity is placed on the sphere of radius $2$. Since we model distant illumination, the sampled colour only depends on direction, and two samples at different locations but in the same direction will sample RENI++ gardner_reni++_2023 at the same point. However, visibility of distant illumination is dependent on location and the intersection of the ray on the sphere of radius $1$ is used to sample our visibility network.
• Figure 4: Visibility of our neural illumination from a point in the scene is implicitly represented via our Directional Distance Field (DDF) which represents the depth to the surface of our scene from any point on the unit sphere. The DDF is a spherical neural field that surrounds our scene at radius $1$. The DDF is fully differentiable allowing gradients obtained from shadowing to inform illumination and geometry.
• Figure 5: Soft visibility function. We plot $\|\mathbf{s}-\mathbf{x}\|-f_\text{DDF}(\mathbf{s},-\mathbf{d})$ on the $x$-axis versus $V(\mathbf{x},\mathbf{d})$ on the $y$-axis. When ground truth distance is significantly smaller than the threshold $\epsilon$, we assign a visibility of 1. When significantly larger, we infer an occlusion and a visibility of 0. In the vicinity of $\epsilon$ we smoothly transition from visible to non-visible with a steepness controlled by $\eta$.