Full Dynamic Range Sky-Modelling For Image Based Lighting

TL;DR

Icarus is an all-weather sky-model capable of learning the exposure range of Full Dynamic Range physically captured outdoor imagery, and illuminates scenes with unprecedented accuracy, photorealism, lighting directionality (shadows), and tones in Image Based Lightning (IBL).

Abstract

Accurate environment maps are a key component to modelling real-world outdoor scenes. They enable captivating visual arts, immersive virtual reality and a wide range of scientific and engineering applications. To alleviate the burden of physical-capture, physically-simulation and volumetric rendering, sky-models have been proposed as fast, flexible, and cost-saving alternatives. In recent years, sky-models have been extended through deep learning to be more comprehensive and inclusive of cloud formations, but recent work has demonstrated these models fall short in faithfully recreating accurate and photorealistic natural skies. Particularly at higher resolutions, DNN sky-models struggle to accurately model the 14EV+ class-imbalanced solar region, resulting in poor visual quality and scenes illuminated with skewed light transmission, shadows and tones. In this work, we propose Icarus, an all-weather sky-model capable of learning the exposure range of Full Dynamic Range (FDR) physically captured outdoor imagery. Our model allows conditional generation of environment maps with intuitive user-positioning of solar and cloud formations, and extends on current state-of-the-art to enable user-controlled texturing of atmospheric formations. Through our evaluation, we demonstrate Icarus is interchangeable with FDR physically captured outdoor imagery or parametric sky-models, and illuminates scenes with unprecedented accuracy, photorealism, lighting directionality (shadows), and tones in Image Based Lightning (IBL).

Figures (45)

• Figure 1: Scenes rendered per $512^2$ environment maps. [Centre] Our DNN sky-model (Fig.1b, per RGB-Icarus with $f_{\text{\tiny Roberson}}$ fusion) recreates the illumination, tones and light transmission of real-world Full Dynamic Range imagery (Fig.1a, FDR ground truth). Icarus accurately models solar illumination for unprecedented lighting directionality (shadows, Fig.1c). [Border] Icarus enables intuitive user-control over positioning and styling of solar and atmospheric formations.
• Figure 2: Each column illustrates the impact of an incremental clipping intensity with exposure equalization to the 14.2EV FDR ground truth. Though these partial-capture environment maps are visually unaltered when clipped from FDR 14.2EV to LDR 1EV, renderings exhibit lost illuminance ($\oiint_I$), altering tones, shadows and light transmission (sun transmission; black glass orb). See \ref{['sec:methodology']} for definition of EV and $\oiint_i$.
• Figure 3: Comparison of AllSky and our model (Icarus) to mitigation of solar modelling by substitution of a parametric Hošek-Wilkie sun (HW HOSEK_13SunDEEPCLOUDS_22) and manual parametric boosting of the HDR environment map (Boosted text2light; $\gamma$=0.5, $\beta$=2, $\rho$=6). Adding an HW sun (LDR+HW Sun) skips atmospheric attenuation and allows the sun to 'pierce' through clouds to create strong shadows. Boosting an HDR image (Boosted LDR) is subject to weather-dependent parameter selection. If the sun is obstructed, boosting is prone to over-exposing and producing unpredictable shadows. Both mitigation strategies alter the perceived tones in IBL renderings (lambertian planar surface), but AllSky and our model (Icarus with $f_{\text{\tiny Robertson}}$ fusion, right column) accurately model real-world FDR illumination for photorealistic IBL renderings. \ref{['fn:mitagation_complete_grids']}
• Figure 4: Visual representation an LDR exposure bracket normalized to HDR-space by \ref{['eq:normalize_ldr_exposure']}. Each LDR exposure is a 'candle-stick' where upper- and lower-limits illustrate min/max HDR intensities for $\Check{I}$ clipped to $[0,1]$ and the body min/max HDR intensities for $\Check{I}$ clipped to $[\epsilon, 1-\epsilon]$ for $\epsilon=\underline{\epsilon}=\overline{\epsilon}=\frac{1}{255}$. Markers ($\circ$) indicate the HDR illumination intensity of an LDR exposure value of 0.5. Insufficient overlap and/or gaps between LDR exposures should be avoided during exposure selection.
• Figure 5: Icarus training architecture. Style-mixer (m) allows for selective training of the RGB-style encoder and RND-style mapper. The Generator (G) and Decoder (D) develop an affinity to the selected style-code source. The decoder outputs an LDR bracket $\{\check{\mathcal{I}}_n\}^N$ which is evaluated per-exposure ($\check{\mathcal{I}}_n$) by the LDR-discriminator $\mathcal{\check{D}}$ to produce loss $\mathcal{L}_{ldr}$ and as bracket by the HDR-discriminator $\mathcal{\hat{D}}$ to produce loss $\mathcal{L}_{hdr}$. Supervised loss $\mathcal{L}_s$ amalgamates per-exposure class-selective losses ($sL_1$) applied to low-variability classes (i.e. border, skydome) and optional per-exposure LPIPS with RGB-style reconstruction tasks.