Learning to Rasterize Differentiably

TL;DR

This work addresses the instability of differentiable rasterization caused by hand-tuned softness functions. It introduces a meta-learning pipeline that jointly optimizes a tunable soft rasterizer, parameterized by an MLP, across diverse inverse-rendering tasks using image-based losses. The results show that the learned softness outperforms fixed distributions and transfers to unseen tasks, while incurring minimal computational overhead, achieving competitive single-view 3D reconstruction performance. The approach automates softness selection, enhances robustness to discontinuities, and offers a pathway to broader applications beyond differentiable rendering.

Abstract

Differentiable rasterization changes the standard formulation of primitive rasterization -- by enabling gradient flow from a pixel to its underlying triangles -- using distribution functions in different stages of rendering, creating a "soft" version of the original rasterizer. However, choosing the optimal softening function that ensures the best performance and convergence to a desired goal requires trial and error. Previous work has analyzed and compared several combinations of softening. In this work, we take it a step further and, instead of making a combinatorial choice of softening operations, parameterize the continuous space of common softening operations. We study meta-learning tunable softness functions over a set of inverse rendering tasks (2D and 3D shape, pose and occlusion) so it generalizes to new and unseen differentiable rendering tasks with optimal softness.

Figures (6)

• Figure 1: Among the continuously differentiable rasterization renderers, we identify the one most suited to solving a family of inverse rendering tasks.
• Figure 2: Meta-learning. Meta-optimization consists of two training loops to jointly optimize the scene parameters for one task instance (vertical) and the renderer parameters across many task instances (horizontal). All columns represent the same task category of changing two triangles' positions to match the reference image in the last row. At test time, the optimal renderer is good at solving unseen tasks, as shown in the rightmost column. This is, because, towards the end of meta-training, the optimization itself mimics the reference closely. The top shows the soft functions used to render one column: a soft depth step makes the triangles transparent, and a soft edge function makes the edges blurry.
• Figure 3: Edge functions for different tasks. We visualize the meta-learned parameters in two sets of tasks. In 3D shape (a) and Pose (b), distinct MLPs are trained for each angle, shown in different shades of blue. For 2D shape (c) and Occlusion (d), the fixed viewpoint necessitates a single MLP, which can be compared to other CDFs with grid-searched softness. The same parameters can be used for more complex scenes (i.e. Transfer tasks) without re-meta-training.
• Figure 4: Every subplot shows the convergence of one inverse rendering task according to one metric where different colors represent different methods. Within each subplot the vertical axis is loss, so less is better (log scale). The horizontal axis is optimization iterations. The first two columns show a training variant, the last two columns show a transfer condition. In each horizontal pair, the first plot is the image-based metric, the second one is the parameter error.
• Figure 5: Results of different methods for the 2D Shape and 3D Shape task. Every pair of rows is one task. The first two pairs are 2D tasks, the second two pairs are 3D tasks. Every even rows show a rendering of the result upon convergence, except for the first column, where we show the initialization (random tris in 2D tasks and a sphere in 3D tasks). Every odd row, shows the error image of that, except for the first column, which shows the reference, the target. A successful optimization would have a black error image and a result that looks similar to the reference. Please refer to the supplementary materials for more analysis.