Rasterized Edge Gradients: Handling Discontinuities Differentiably

TL;DR

EdgeGrad introduces a micro-edge based differentiable rasterization framework that preserves forward rasterization but enables accurate gradient computation at visibility discontinuities, including triangle boundaries, occlusions, and geometry intersections. The key derivation yields a closed-form edge-gradient for a boundary between two pixels, $\frac{\partial L}{\partial p_{AB}}=\tfrac{1}{2}(\frac{\partial L}{\partial I_A}+\frac{\partial L}{\partial I_B})(I_A-I_B)+\Omega$, plus a mechanism to scatter these gradients to geometry via $\frac{\partial p}{\partial r}$, accommodating overhangs and intersections. The method is implemented in PyTorch/CUDA and validated on synthetic and Blender-based datasets, showing comparable or superior gradient accuracy to finite differences, especially for intersecting geometries, while delivering significant speedups. It enables robust, automatic mesh-based inverse rendering, demonstrated in dynamic head scenes with mouth interior fitting guided by segmentation supervision, though it omits transparency effects and global illumination.

Abstract

Computing the gradients of a rendering process is paramount for diverse applications in computer vision and graphics. However, accurate computation of these gradients is challenging due to discontinuities and rendering approximations, particularly for surface-based representations and rasterization-based rendering. We present a novel method for computing gradients at visibility discontinuities for rasterization-based differentiable renderers. Our method elegantly simplifies the traditionally complex problem through a carefully designed approximation strategy, allowing for a straightforward, effective, and performant solution. We introduce a novel concept of micro-edges, which allows us to treat the rasterized images as outcomes of a differentiable, continuous process aligned with the inherently non-differentiable, discrete-pixel rasterization. This technique eliminates the necessity for rendering approximations or other modifications to the forward pass, preserving the integrity of the rendered image, which makes it applicable to rasterized masks, depth, and normals images where filtering is prohibitive. Utilizing micro-edges simplifies gradient interpretation at discontinuities and enables handling of geometry intersections, offering an advantage over the prior art. We showcase our method in dynamic human head scene reconstruction, demonstrating effective handling of camera images and segmentation masks.

Figures (21)

• Figure 1: Micro-edges and Pixel Pair. The figure illustrates construction of the micro-edges and the change in pixel intensities A and B with the edge position $p$ movement, highlighting the C1 discontinuity at $p=0$, absent in average intensity.
• Figure 1: Accuracy of backward gradients. Relative error, % ($\downarrow$). This table shows relative errors in backward gradient computations for test scenes. Second and third scenes include geometry intersections, emphasizing our method's advantage in managing these complexities.
• Figure 2: Gradient computation flow. The figure illustrates the backward propagation process, where gradients from the loss function are traced back and scattered to the fragments.
• Figure 3: Edge classification. The accompanying figure provides a schematic representation of our edge classification process. Edges are categorized into four types: no edge, adjacent primitives, overhanging primitives, and intersecting primitives.
• Figure 4: Geometry intersection. A geometrical derivation for the partial derivative $\frac{\partial p}{ \partial r}$, essential for converting edge-related gradients to fragment-related gradients in cases where edges are formed by geometry intersections. This shows the x-z plane, where the render camera is viewing the scene along the z axis so that edges are parallel to the $y$ axis (and perpendicular to the drawing). Projections of both intersecting primitives are transformed into two lines.