Evolutive Rendering Models

TL;DR

The paper introduces Evolutive Rendering Models (ERM), a framework that replaces hand-crafted heuristic rendering components with differentiable, evolvable elements that adapt during rendering. It develops three core components—Evolutive Gauge Transformation, Evolutive Ray Sampling, and Evolutive Primitive Organization—each designed to be differentiable and guided by the final rendering objective through a relay learning mechanism that stabilizes early training. The authors provide gradient analyses and extensive experiments across static, dynamic, and generative tasks, showing improvements in PSNR, SSIM, LPIPS, and even FID, while enabling new capabilities like UV mapping and cross-scene generalization. This work demonstrates that learnable, evolutive components can significantly enhance rendering performance, efficiency, and flexibility across diverse representation types and rendering paradigms.

Abstract

The landscape of computer graphics has undergone significant transformations with the recent advances of differentiable rendering models. These rendering models often rely on heuristic designs that may not fully align with the final rendering objectives. We address this gap by pioneering \textit{evolutive rendering models}, a methodology where rendering models possess the ability to evolve and adapt dynamically throughout the rendering process. In particular, we present a comprehensive learning framework that enables the optimization of three principal rendering elements, including the gauge transformations, the ray sampling mechanisms, and the primitive organization. Central to this framework is the development of differentiable versions of these rendering elements, allowing for effective gradient backpropagation from the final rendering objectives. A detailed analysis of gradient characteristics is performed to facilitate a stable and goal-oriented elements evolution. Our extensive experiments demonstrate the large potential of evolutive rendering models for enhancing the rendering performance across various domains, including static and dynamic scene representations, generative modeling, and texture mapping.

Figures (14)

• Figure 1: Evolutive Rendering Models covers three principal rendering elements: gauge transformation, ray sampling and primitive organization. All three elements can be applied in volume rendering, while splatting only employs evolutive gauge transformation and primitive organization.
• Figure 2: A conceptual illustration of rendering elements, including ray sampling in volume rendering, primitive organization in point-based rendering, and gauge transformations. Volumetric and point-based rendering can be performed in a unified manner: accumulating or blending discrete points relevant to the given ray.
• Figure 3: Motivation of evolutive gauge transformation. In this example, instead of mapping the 3D euclidean space to the 2D plane by orthogonal projection (left), we learn a more flexible and adaptive mapping (right).
• Figure 4: The illustration of predefined (upper) and evolutive (lower) gauge transformation. We implement the evolutive transformation by predicting an offset to the pre-defined transformation.
• Figure 5: Differentiable sampling with piecewise linear approximation.