A Generalizable Light Transport 3D Embedding for Global Illumination

TL;DR

This work introduces a generalizable 3D light transport embedding (LTM) that directly learns from 3D scene configurations to predict global illumination across unseen indoor environments. A point-based intermediate representation and a transformer-based Light Transport Encoder capture long-range scene interactions, while a Local Query Decoder aggregates neighboring latent codes to produce irradiance or radiance at query points, enabling view- and resolution-independent GI without rasterized or path-traced cues. The authors demonstrate diffuse GI generalization across diverse scenes, show that the learned embedding can be quickly repurposed for glossy radiance fields and used to jump-start path guiding, and release a large indoor dataset to support reproducibility. This approach pushes toward integrating learned priors into rendering pipelines with potential benefits for efficiency and consistency across views, materials, and lighting configurations. Overall, the method offers a scalable, differentiable alternative to traditional GI that can adapt to multiple rendering tasks with limited fine-tuning.

Abstract

Global illumination (GI) is essential for realistic rendering but remains computationally expensive due to the complexity of simulating indirect light transport. Recent neural methods have mainly relied on per-scene optimization, sometimes extended to handle changes in camera or geometry. Efforts toward cross-scene generalization have largely stayed in 2D screen space, such as neural denoising or G-buffer based GI prediction, which often suffer from view inconsistency and limited spatial understanding. We propose a generalizable 3D light transport embedding that approximates global illumination directly from 3D scene configurations, without using rasterized or path-traced cues. Each scene is represented as a point cloud with geometric and material features. A scalable transformer models global point-to-point interactions to encode these features into neural primitives. At render time, each query point retrieves nearby primitives via nearest-neighbor search and aggregates their latent features through cross-attention to predict the desired rendering quantity. We demonstrate results on diffuse global illumination prediction across diverse indoor scenes with varying layouts, geometry, and materials. The embedding trained for irradiance estimation can be quickly adapted to new rendering tasks with limited fine-tuning. We also present preliminary results for spatial-directional radiance field estimation for glossy materials and show how the normalized field can accelerate unbiased path guiding. This approach highlights a path toward integrating learned priors into rendering pipelines without explicit ray-traced illumination cues.

Figures (16)

• Figure 1: Visualization of attention for a bedroom floorplan with a hallway. We show the attention heatmaps for two different focus scene points. The points are shaded using importance as the alpha value whereas the top 200 points with the highest attention scores are larger. The attention is extracted from our light transport encoder. The blue shaded square denotes the light source. In (b), we see four groups of scene points having the largest impact on the focus point (highlighted in blue square and orange clusters), which likely corresponds to 4 bounces of ray tracing. In (c), the focus point is on the wall of the hallway where the light source is occluded, so no points on the light source contribute much. Points with high attention scores are mostly located on different walls in the hallway, also forming three groups (highlighted in orange clusters). This serves as a visual hint for our analogy between the light transport operator and the attention mechanism in \ref{['subsec:analogy']}.
• Figure 2: Our pipeline. Our method first converts the original 3D assets of each scene into a point cloud with associated features (on the left). The scenes are then encoded into a light transport embedding, consisting of latent codes (depicted as small blocks within the dashed circle) anchored at the scene point positions. At render time (center), each query point gathers local latent codes via k-nearest neighbors and feeds them together with the query point attributes into a target-specific query decoder, of which the core is a cross-attention.
• Figure 3: Network architecture for Light Transport Encoder and Local Query Irradiance Decoder.
• Figure 4: Dataset overview. From left to right, examples of: a) generated floor plans, b) generated untextured meshes for a floor plan, c) generated texture atlases for mesh objects, d) textured mesh objects, e) textured meshes converted to scene points directly fed into our pipeline.
• Figure 5: Irradiance prediction comparison with baseline representatives across six diverse floor-plans under varying illumination conditions. We primarily highlight the substantial qualitative visual improvements while showing SSIM as a quantitative reference. We evaluate our model on a held-out test set to demonstrate its generalization capability. Notably, no per-scene training is performed—unlike prior neural global illumination approaches. For Deep Shading, which fails to generalize to new scenes, we include the test scenes in their training set to show their overfitting results here. Additionally, we extend the original implementation by hermosilla2019deep to handle our more complex scenes, as their original setup is limited to simple object-level inputs and performs poorly on our dataset (see \ref{['sec:diffuse_GI']}). The first column shows the corresponding albedo to provide readers with context for the color of the indirect lighting, while we avoid showing albedo-modulated images to maintain a clear focus on the quality of the predicted irradiance without distraction.