SIR: Multi-view Inverse Rendering with Decomposable Shadow Under Indoor Intense Lighting

TL;DR

SIR tackles indoor inverse rendering by explicitly decomposing differentiable shadows from albedo using a three-stage material estimation within an SDF-based HDR radiance field. A three-phase training scheme separates geometry, lighting, shadows, and SVBRDF, aided by an irradiance field and shadowMLPs to model hard and soft shadows and BRDF regularization. The approach yields improved SVBRDF quality, more accurate shadow recovery, and enables high-fidelity free-view relighting, object insertion, and material replacement, demonstrated on synthetic and real indoor datasets. This work advances practical indoor scene editing by achieving physically plausible shading and robust shadow decomposition under unknown lighting, with publicly available code and data.

Abstract

We propose SIR, an efficient method to decompose differentiable shadows for inverse rendering on indoor scenes using multi-view data, addressing the challenges in accurately decomposing the materials and lighting conditions. Unlike previous methods that struggle with shadow fidelity in complex lighting environments, our approach explicitly learns shadows for enhanced realism in material estimation under unknown light positions. Utilizing posed HDR images as input, SIR employs an SDF-based neural radiance field for comprehensive scene representation. Then, SIR integrates a shadow term with a three-stage material estimation approach to improve SVBRDF quality. Specifically, SIR is designed to learn a differentiable shadow, complemented by BRDF regularization, to optimize inverse rendering accuracy. Extensive experiments on both synthetic and real-world indoor scenes demonstrate the superior performance of SIR over existing methods in both quantitative metrics and qualitative analysis. The significant decomposing ability of SIR enables sophisticated editing capabilities like free-view relighting, object insertion, and material replacement. The code and data are available at https://xiaokangwei.github.io/SIR/.

Figures (9)

• Figure 1: Given posed multi-view HDR images of an indoor scene, SIR successfully disentangle the scene appearance into inherent attributes, which can produce convincing results for several free-viewpoint editing applications.
• Figure 2: The pipeline consists of three phases: 1) We sample a ray with direction $\bm{v}$ and spatial point $\textbf{x}$ from the given posed HDR images. The geometry network $f_d$ learns the signed distance $d$ to obtain surface point $\hat{\textbf{x}}$, and the HDR-radiance network $f_c$ learns radiance $\hat{C}$. 2) We obtain the diffuse incoming lighting $L_{i,d}$ by integrating incident radiance from environment maps $E$ for learning irradiance $I$. 3) Hard shadow $S_{hard}$ is learned using $\Theta_h$ with pseudo ground truth $\xi$. We then initialize the parameters of $\Theta_s$ using the optimized parameters of $\Theta_h$. Instance-level BRDF regularizers are applied, and the whole rendering equation is optimized to update $\hat{A}$, $\hat{R}$, and $S_{soft}$.
• Figure 3: Qualitative results of all methods on two datasets (Syn: Restroom, Real: Office).
• Figure 4: Qualitative results of ablation study (Kitchen in synthetic dataset).
• Figure 5: Qualitative results of scene editing using our SIR on synthetic dataset.