RISE-SDF: a Relightable Information-Shared Signed Distance Field for Glossy Object Inverse Rendering

TL;DR

RISE-SDF addresses the challenge of inverse rendering for glossy objects by proposing a two-stage factorization that jointly optimizes geometry and appearance. The first stage builds a reflection-aware radiance field on a neural SDF geometry representation, while the second stage learns material parameters and lighting with an information-sharing MLP framework and a physically based split-sum rendering strategy, including a second split-sum for indirect illumination during relighting. A key novelty is the information-sharing mechanism between geometry-driven radiance and BRDF-based factors, plus an Indirect Illumination MLP and a second split-sum relighting pass, which yield state-of-the-art results in both geometry/material reconstruction and relighting on glossy objects. The authors also introduce a Shiny Inverse Rendering Synthetic Dataset with ground-truth BRDF parameters and relighting across multiple env maps, enabling robust evaluation of glossy inverse rendering systems. Overall, the approach delivers high-quality novel-view synthesis and robust relighting for glossy materials with improved efficiency and stability compared to prior methods, advancing practical asset creation for games and visual effects.

Abstract

In this paper, we propose a novel end-to-end relightable neural inverse rendering system that achieves high-quality reconstruction of geometry and material properties, thus enabling high-quality relighting. The cornerstone of our method is a two-stage approach for learning a better factorization of scene parameters. In the first stage, we develop a reflection-aware radiance field using a neural signed distance field (SDF) as the geometry representation and deploy an MLP (multilayer perceptron) to estimate indirect illumination. In the second stage, we introduce a novel information-sharing network structure to jointly learn the radiance field and the physically based factorization of the scene. For the physically based factorization, to reduce the noise caused by Monte Carlo sampling, we apply a split-sum approximation with a simplified Disney BRDF and cube mipmap as the environment light representation. In the relighting phase, to enhance the quality of indirect illumination, we propose a second split-sum algorithm to trace secondary rays under the split-sum rendering framework. Furthermore, there is no dataset or protocol available to quantitatively evaluate the inverse rendering performance for glossy objects. To assess the quality of material reconstruction and relighting, we have created a new dataset with ground truth BRDF parameters and relighting results. Our experiments demonstrate that our algorithm achieves state-of-the-art performance in inverse rendering and relighting, with particularly strong results in the reconstruction of highly reflective objects.

Figures (13)

• Figure 1: RISE-SDF. We present RISE-SDF, a method for reconstructing the geometry and material of glossy objects while achieving high-quality relighting. Our results, compared with the state-of-the-art method liu2023nero, show superior albedo and roughness estimation with significantly less training time. As an end-to-end relightable model, our algorithm generates high-quality relighting images without noise or aliasing.
• Figure 2: Our pipeline. The colors of the features in the figure indicate different feature concatenation combinations. (1) Given the location $\boldsymbol{x}_i$, the progressive hash grid with the geometry MLP predicts the geometry feature $\beta_i$ and the corresponding volume rendering weight $w_i$ via Eq. \ref{['eqn:vol_render_neus']} and \ref{['eqn:alpha_occ']}. (2) For the color representation, separate networks predict per-sample albedo $\boldsymbol{a_i}$, metallic $m_i$, and roughness $\rho_i$. They share the information between the direct volume rendering pipeline (black arrow) and the physically based rendering pipeline (red arrow). (3) The per-sample values (all except the blue value) are rendered via volume rendering. Then we compute the expected surface intersection $\boldsymbol{\hat{x}}$ and trace another ray to compute the occlusion probability $O$. Finally, the direct and indirect colors are blended via $O$.
• Figure 3: Different Illumination Inference Method. (a) During training and relighting, we use the first split-sum to compute the direct illumination. (b) During training, we use an MLP to predict indirect illumination and blend with direct illumination using the occlusion probability. (c) During relighting, we use a second split-sum with one additional ray bounce to compute the indirect illumination.
• Figure 4: Qualitative comparisons on the helmet scene. From top to bottom: relighting, material. We can observe that other methods either have blurry, color-shifted results or aliasing, noisy effect under unseen illumination. And our material estimation also outperform other baselines.
• Figure 5: Ablation study for indirect illumination.