Many-Worlds Inverse Rendering

TL;DR

This work introduces a many-worlds inverse rendering framework that mitigates discontinuous visibility during geometry reconstruction by differentiating a volumetric perturbation of a surface rather than a local surface change. The method defines an extended parameter space with an occupancy field $\alpha(\mathbf{x})$ and an orientation field $\beta(\mathbf{x})$, centered on a background surface $\bar{S}$ and a perturbation distribution $S$, and derives a many-worlds derivative transport that aggregates derivatives from all hypothetical surface patches along a ray. By formulating a primal rendering that averages perturbation contributions with a scaling factor $1/s$, the approach achieves simpler implementation and faster convergence compared to silhouette-sampling or volumetric methods, while yielding a final mesh ready for relighting. The framework unifies shading and occlusion derivatives in the extended domain and demonstrates strong performance on multi-view object reconstructions, including challenging specular and obstacle-rich scenes, while acknowledging limitations like exterior-only perturbations and opportunities for richer directional perturbations in future work.

Abstract

Discontinuous visibility changes remain a major bottleneck when optimizing surfaces within a physically-based inverse renderer. Many previous works have proposed sophisticated algorithms and data structures to sample visibility silhouettes more efficiently. Our work presents another solution: instead of differentiating a tentative surface locally, we differentiate a volumetric perturbation of a surface. We refer this as a many-worlds representation because it models a non-interacting superposition of conflicting explanations (worlds) of the input dataset. Each world is optically isolated from others, leading to a new transport law that distinguishes our method from prior work based on exponential random media. The resulting Monte Carlo algorithm is simpler and more efficient than prior methods. We demonstrate that our method promotes rapid convergence, both in terms of the total iteration count and the cost per iteration.

Figures (16)

• Figure 1: Many-worlds derivative transport. The propagated derivative at distance $t$ is only weighted by the local radiance difference and the local occupancy respectively (\ref{['eqn:manyworlds-deriv']}). In contrast to exponential or non-exponential (e.g., linear vicini2021non) volumes, the notion of transmittance disappears as it models nonsensical inter-world shadowing.
• Figure 2: Occupancy and orientation fields. The above images visualize the contents of an occupancy ($\alpha$) and orientation ($\bm{\beta}$) field following an optimization. The former models the probability of a surface existing at a position, while the latter assigns normal directions.
• Figure 3: Importance of the orientation field. We compare reconstructions done with and without an orientation field $\bm{\beta}$. The top row without $\bm{\beta}$ uses an isotropic normal distribution. A lack of orientation information dramatically slows down convergence and produces incorrect meshes.
• Figure 4: Our method refines a surface $\bar{S}$ by optimizing a distribution $S$ of possible surfaces. Each hypothetical surface patch drawn from $S$ is independently adjusted to improve the matching between $\bar{S}$ and the target image. To define the behavior of possible surfaces in space, $S$ is parameterized by an occupancy field $\alpha(\mathbf{x})$ and an orientation field $\bm{\beta}(\mathbf{x})$.
• Figure 5: Multi-view geometry reconstruction. For the Deer scene, all $8$ views are behind the object and we only see the front side in the mirror. For Polyhedra, Neptune and Fertility, the object is inside a spherical or cubic smooth glass container. The material is known during optimization: the Dragon has a rough gold material, Polyhedra and Neptune are made of copper oxide, and others are diffuse.