Environment Maps Editing using Inverse Rendering and Adversarial Implicit Functions

TL;DR

This work proposes to model the optimized environment map with a new variant of implicit neural representations able to handle HDR images, trained with adversarial perturbations over the weights to ensure smooth changes in the output when it receives gradients from the inverse rendering.

Abstract

Editing High Dynamic Range (HDR) environment maps using an inverse differentiable rendering architecture is a complex inverse problem due to the sparsity of relevant pixels and the challenges in balancing light sources and background. The pixels illuminating the objects are a small fraction of the total image, leading to noise and convergence issues when the optimization directly involves pixel values. HDR images, with pixel values beyond the typical Standard Dynamic Range (SDR), pose additional challenges. Higher learning rates corrupt the background during optimization, while lower learning rates fail to manipulate light sources. Our work introduces a novel method for editing HDR environment maps using a differentiable rendering, addressing sparsity and variance between values. Instead of introducing strong priors that extract the relevant HDR pixels and separate the light sources, or using tricks such as optimizing the HDR image in the log space, we propose to model the optimized environment map with a new variant of implicit neural representations able to handle HDR images. The neural representation is trained with adversarial perturbations over the weights to ensure smooth changes in the output when it receives gradients from the inverse rendering. In this way, we obtain novel and cheap environment maps without relying on latent spaces of expensive generative models, maintaining the original visual consistency. Experimental results demonstrate the method's effectiveness in reconstructing the desired lighting effects while preserving the fidelity of the map and reflections on objects in the scene. Our approach can pave the way to interesting tasks, such as estimating a new environment map given a rendering with novel light sources, maintaining the initial perceptual features, and enabling brush stroke-based editing of existing environment maps.

Figures (9)

• Figure 1: We start from an initial environment map $\envmap$ that we parameterize using an implicit function via a robust neural network $f_\theta(\bx)$. We optimize the new environment map by receiving gradients through an inverse differentiable rendering that matches the rendered image $\I{rendered}$ to a target $\I{target}$. The target is generated by an unknown environment map $\envmap_{\text{target}}$; in practical settings, $\I{target}$ could be created by roughly painting possible shades from $\I{rendered}$ so to "find" the generating environment map.
• Figure 2: Inverse rendering in the proposed SIREN HDR
• Figure 3: SIREN HDR can represent HDR images with higher fidelity and better pixel intensity distribution, compared to the baseline architecture in sitzmann2019siren.
• Figure 4: Effects of applying the same small random perturbation to implicit functions. The random perturbation is $\mbf{W}^{\star}\sim \alpha\mathcal{N}(0,1)$ with $\alpha$ set to $1\times 10^{-3}$ and is applied in the weights space. From left to right: the original environment map, the rendering of SIREN HDR, the rendering of R-SIREN HDR, the perturbation applied to SIREN HDR, and, finally, the perturbation applied to R-SIREN HDR. In this last case, perturbation of the weights induces naturally looking maps.
• Figure 5: Results over many SIREN training strategies and experiment setups. The initial rendering is obtained with the initial environment map. The goal is to run the optimization to reach the target's lighting conditions. Note that the target environment map is unknown during the optimization.