A Survey on Physics-based Differentiable Rendering

TL;DR

A comprehensive overview of the current state of the art in physics-based differentiable rendering is provided, focusing on recent advances in general differentiable rendering theory, Monte Carlo sampling strategy, and computational efficiency.

Abstract

Physics-based differentiable rendering has emerged as a powerful technique in computer graphics and vision, with a broad range of applications in solving inverse rendering tasks. At its core, differentiable rendering enables the computation of gradients with respect to scene parameters, allowing optimization-based approaches to solve various problems. Over the past few years, significant advancements have been made in both the underlying theory and the practical implementations of differentiable rendering algorithms. In this report, we provide a comprehensive overview of the current state of the art in physics-based differentiable rendering, focusing on recent advances in general differentiable rendering theory, Monte Carlo sampling strategy, and computational efficiency.

Figures (8)

• Figure 1: Differential spherical integral: in this simple 2D scene, object A is moving with $\pi$, occluding part of the yellow light source. Considering $x_1$ as the shading point, $x^\mathrm{B}$ becomes a discontinuity boundary, and its projection on the unit sphere $\mathbb{S}^2$ becomes part of $\Delta\mathbb{S}^2$. The gradient caused by the motion of $\Delta\mathbb{S}^2$ can not be estimated by the interior integral, and needs to be treated separately in the boundary integral.
• Figure 2: Differential path integral: in this simple 2D scene, object A is moving with $\pi$. (a) The light path is an ordinary light path that does not involve a boundary segment. (b) The light path is a boundary light path containing a boundary segment $(x_2, x_3)$, and therefore the material form of the light path belongs to $\partial \hat{\Omega}$.
• Figure 3: PSDR for implicit surfaces: (a) primary boundaries can be viewed as the intersected points (in 3D, curves) of $\phi$ and $\psi$. (b) To sample primary boundaries, we cast rays from $q_i, \dots, q_i$ until the ray intersects with $\mathcal{S}$ or $q_i$ reaches the other end. Reference: Figure 5 in Zhou:2024:PSDR-SDF.
• Figure 4: Constructing warp field: (a) a discontinuous warp field constructed by tracing auxiliary rays. (b) A continuous warp field smoothed by the convolution kernel $w$. Note that the values on boundary remain intact. Reference: Figure 6 in bangaru:2020:warpedsampling.
• Figure 5: Warped-area reparameterization for differential path integral: this scene illustrates the boundary segment $(x_K, x_{K+1})$ and the corresponding discontinuity curve $\Delta \mathcal{B}_K$. After applying divergence theorem, the domain of integration is converted to the whole plane $\mathcal{B}_K$. Reference: Figure 6 in Xu:2023:PSDR-WAS.