RNA: Relightable Neural Assets

TL;DR

This work tackles the high cost and engineering burden of rendering complex shaded 3D assets by introducing relightable neural assets that encode light transport directly within the asset. The approach preserves explicit geometry and augments it with a triplane feature grid and an MLP that, given shading position, view, and light directions, outputs a neural transport function ${\mathcal{T}}$ for fast, relightable shading under Monte Carlo path tracing. The authors provide an end-to-end data generation and training pipeline (400 camera views, per-pixel random lighting, 90–120 minutes on an A100) and demonstrate integration into production renderers with significant speedups (up to 200× for hair) while closely matching ground-truth path tracing. Key contributions include a compact neural representation for complex shading on both surface and fiber assets, an efficient data-and-training regime, and a practical rendering pipeline that yields high-fidelity relighting across diverse lighting environments, enabling scalable use in real-time or production contexts. This work advances the practicality of neural rendering for production pipelines by decoupling complex shading from the renderer while maintaining full light transport compatibility and near-photorealistic results, including near-field fiber effects.

Abstract

High-fidelity 3D assets with materials composed of fibers (including hair), complex layered material shaders, or fine scattering geometry are ubiquitous in high-end realistic rendering applications. Rendering such models is computationally expensive due to heavy shaders and long scattering paths. Moreover, implementing the shading and scattering models is non-trivial and has to be done not only in the 3D content authoring software (which is necessarily complex), but also in all downstream rendering solutions. For example, web and mobile viewers for complex 3D assets are desirable, but frequently cannot support the full shading complexity allowed by the authoring application. Our goal is to design a neural representation for 3D assets with complex shading that supports full relightability and full integration into existing renderers. We provide an end-to-end shading solution at the first intersection of a ray with the underlying geometry. All shading and scattering is precomputed and included in the neural asset; no multiple scattering paths need to be traced, and no complex shading models need to be implemented to render our assets, beyond a single neural architecture. We combine an MLP decoder with a feature grid. Shading consists of querying a feature vector, followed by an MLP evaluation producing the final reflectance value. Our method provides high-fidelity shading, close to the ground-truth Monte Carlo estimate even at close-up views. We believe our neural assets could be used in practical renderers, providing significant speed-ups and simplifying renderer implementations.

Figures (16)

• Figure 1: Overview of the pipeline. On the left, we illustrate our triplane representation, consisting of XY, XZ and YZ planes each with 8 feature channels and a default resolution of $512 \times 512$. The feature vectors queried from the triplane representation are summed and passed into an MLP, along with additional properties. We show two configuration variants of relightable neural asset pipelines designed for surface rendering (middle) and fiber rendering (right), respectively. The main difference is that surfaces use a normal input, while fibers use tangent and cross-section offset. Both variants output two colors, one of which will be picked according to the visibility at render time.
• Figure 2: Visualization of the triplane representation. We show the triplane representation for the subsurface lego asset, visualizing the first channel of XY, YZ, and XZ planes, respectively.
• Figure 3: Data generation of our pipeline. (a) We sample camera views around the asset (upper hemisphere) and for each training slice (b) the light directions are randomly sampled for each pixel's ray hit shading point. In (c), we visualize two slices of the training data and AOVs obtained from Blender Cycles path tracer.
• Figure 4: Training convergence plots on translucent lego and blue hair assets. We show the convergence plots for two versions of our model on a surface asset (lego) and a fiber asset (blue hair). The Lego Small and Hair Small are trained with a small variant of our neural model with reduced MLP size for real-time rendering in path tracer, at the cost of compromising some accuracy.
• Figure 5: Path tracer integration on surface-based asset. We integrate surface models into a production path tracer on both CPU and GPU rendering. Our model significantly simplifies the shader implementation and improves the rendering performance. In contrast, blender path tracing exhibits severe Monte Carlo noise at equal rendering time budget.