A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

TL;DR

This work exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes and represents appearance with a multi‐layer perceptron parameterized on light direction, to enforce multi‐view consistency and overcome inaccuracies.

Abstract

Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic -- but possibly inconsistent -- multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. Project site https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/

Figures (12)

• Figure 1: We use the single-view, multi-illumination dataset of Murmann et al. multilum to train ControlNet controlnet on single view supervised relighting. The network accepts an image (along with its estimated depth map) and a target light direction as input and produces a relit version of the same scene under the desired target lighting.
• Figure 2: Top row: five diffuse sphere rendered by our optimized lighting direction and shading parameters --- the direction is indicated by a blue dot at the point of maximum specular intensity; Bottom row: the corresponding target gray spheres obtained by averaging the diffuse spheres captured in all spheres. We found the lighting directions by minimizing the $L_1$ distance between the top and bottom row.
• Figure 3: Importance of post-relighting color and contrast adjustments. Left: input image. Middle: naive ControlNet relighting; the bottle has the wrong color and the contrast is poor. Right: our relighting after training with flawed and after color-matching the input.
• Figure 4: Importance of conserving edge sharpness when relighting. Left: input image. Middle: naive ControlNet relighting; note how the edges do not match the input and how the text is illegible. Right: our final relighting after fine-tuning the conditonal decoder network from zhu2023designing.
• Figure 5: Relighting results with our light-conditioned ControlNet. From a single input image (left column), the ControlNet can generate realistic relit versions for different target light directions (other columns). Please notice realistic changes in highlights for different light directions (top row), as well as the synthesis of cast shadows (bottom row).