TensoIR: Tensorial Inverse Rendering

TL;DR

This work extends TensoRF, a state-of-the-art approach for radiance field modeling, to estimate scene geometry, surface reflectance, and environment illumination from multi-view images captured under unknown lighting conditions, leading to photo-realistic novel view synthesis and relighting results.

Abstract

We propose TensoIR, a novel inverse rendering approach based on tensor factorization and neural fields. Unlike previous works that use purely MLP-based neural fields, thus suffering from low capacity and high computation costs, we extend TensoRF, a state-of-the-art approach for radiance field modeling, to estimate scene geometry, surface reflectance, and environment illumination from multi-view images captured under unknown lighting conditions. Our approach jointly achieves radiance field reconstruction and physically-based model estimation, leading to photo-realistic novel view synthesis and relighting results. Benefiting from the efficiency and extensibility of the TensoRF-based representation, our method can accurately model secondary shading effects (like shadows and indirect lighting) and generally support input images captured under single or multiple unknown lighting conditions. The low-rank tensor representation allows us to not only achieve fast and compact reconstruction but also better exploit shared information under an arbitrary number of capturing lighting conditions. We demonstrate the superiority of our method to baseline methods qualitatively and quantitatively on various challenging synthetic and real-world scenes.

Figures (7)

• Figure 1: Given multi-view captured images of a real scene (a), our approach -- TensoIR -- is able to achieve high-quality shape and material reconstruction with high-frequency details (b). This allows us to render the scene under novel lighting and viewpoints (c), and also change its material properties (d).
• Figure 2: Overview. We propose a novel inverse rendering approach to reconstruct scene geometry, materials, and unknown natural illumination (as an environment map) from captured images. We reconstruct a scene as a novel representation (Sec. \ref{['sec:representation']}) that uses factorized tensors and multiple MLPs to regress volume density $\sigma$, view-dependent color $c$, normals $\mathbf{n}$, and material properties (i.e. BRDF parameters) $\boldsymbol{\beta}$, enabling both radiance field rendering and physically-based rendering (Sec. \ref{['sec:rendering']}). In particular, we march a camera ray from camera origin $\mathbf{o}$ in viewing direction $\mathbf{d}$, sample points $\mathbf{x}_j$ along the ray, and apply radiance field rendering using the density and view-dependent colors regressed from our representation (Eqn. \ref{['equ:rfrendering']}). We also use the volume rendering weights to determine the surface point $\mathbf{\hat{x}}$ on the ray (Eqn. \ref{['equ:surf']}), at which we perform physically based rendering using the normals and material properties (Eqn. \ref{['equ:pbrender']}). We compute accurate visibility $V$ and indirect lighting $L_{\text{ind}}$ using radiance field rendering by marching secondary rays from the surface point $\mathbf{\hat{x}}$ along sampled incoming light direction $\boldsymbol{\omega}_i$ (Sec. \ref{['sec:indirect']}), enabling accurate physically-based rendering. We supervise both the radiance field rendering $C_{\text{RF}}$ and physically-based rendering $C_{\text{PB}}$ with the captured images in a per-scene optimization for joint scene reconstruction (Sec. \ref{['sec:recon']}).
• Figure 3: We compare normal and albedo reconstruction results between our joint reconstruction model and an ablated model without radiance field rendering during reconstruction. Radiance field reconstruction is crucial for us to achieve good reconstruction with a clean background and reasonable scene geometry.
• Figure 4: We show our computed indirect illumination (b) of the full rendered image (a) and our lighting visibility (c) under two different directional lights.
• Figure 5: Visual comparison against baseline methods. Our method produces inverse rendering results of higher quality with more detailed normals and more accurate albedo, thus leading to more photo-realistic relighting results.