Towards Physically-Based Sky-Modeling For Image Based Lighting

TL;DR

The paper addresses the gap in physically accurate, full dynamic-range sky-modeling for image-based lighting by introducing AllSky, a conditional sky-model trained directly on physically captured FDR HDRIs. It systematically studies input modalities, tone-mapping operators, and learning losses (supervised and unsupervised) to produce environment maps that preserve illumination fidelity while enabling user control over sun and cloud configurations. Through extensive ablations and comparisons against CloudNet, SkyGAN, and Text2Light on the Laval HDR Sky database, AllSky demonstrates state-of-the-art visual quality and illumination metrics, while revealing limitations of current DNN sky-models at higher resolutions and the inadequacy of common evaluation metrics for sky IL fidelity. The work emphasizes the need for metrics that capture Exposure Range and Integrated Illumination and lays out a framework for robust evaluation and conditioning of sky-models in practical IBL pipelines.

Abstract

Accurate environment maps are a key component for rendering photorealistic outdoor scenes with coherent illumination. They enable captivating visual arts, immersive virtual reality, and a wide range of engineering and scientific applications. Recent works have extended sky-models to be more comprehensive and inclusive of cloud formations but, as we demonstrate, existing methods fall short in faithfully recreating natural skies. Though in recent years the visual quality of DNN-generated High Dynamic Range Imagery (HDRI) has greatly improved, the environment maps generated by DNN sky-models do not re-light scenes with the same tones, shadows, and illumination as physically captured HDR imagery. In this work, we demonstrate progress in HDR literature to be tangential to sky-modelling as current works cannot support both photorealism and the 22 f-stops required for the Full Dynamic Range (FDR) of outdoor illumination. We achieve this by proposing AllSky, a flexible all-weather sky-model learned directly from physically captured HDRI which we leverage to study the input modalities, tonemapping, conditioning, and evaluation of sky-models. Per user-controlled positioning of the sun and cloud formations, AllSky expands on current functionality by allowing for intuitive user control over environment maps and achieves state-of-the-art sky-model performance. Through our proposed evaluation, we demonstrate existing DNN sky-models are not interchangeable with physically captured HDRI or parametric sky-models, with current limitations being prohibitive of scalability and accurate illumination in downstream applications

Figures (78)

• Figure 1: IBL renders of AllSky environment maps and AllSky environment maps generated from user-drawn labels.
• Figure 2: Each column illustrates the impact of an incremental clipping of exposure with exposure equalization to the 15EV ground truth. Though the environment-maps are visually unaltered, renderings have altered tones, shadows and light transmission (sun transmission; black glass orb).
• Figure 3: Tone-Mapping operator ($T_m$) compression (left) and respective non-linearity between error ($\delta$=$0.01$) in LDR compressed space and error ($\Delta$) in uncompressed HDR space (right).
• Figure 4: AllSky baseline architecture, accepting arbitrary discrete ($\mathbb{Z}^{+}$) or continuous ($\mathbb{R}$) $c_{i}$-channel input labels. The $c_{o}$-channel output can be evaluated by losses and metrics in $LDR$ compressed space, ${cLDR}$ compressed and clipped LDR-space, and ${HDR}$ inverse-tone-mapped linear space.
• Figure 5: 3D surface of skydome illumination, coloured from low to high intensity. Disk above 3D surface illustrates skydome segmentation. The sun (red spike) is a small subset of pixels whose intensities belittle the remainder of the skydome. HDRDB sample from June 7th, 2016 at 1:54PM LavalHDRdb.