Compact Hadamard Latent Codes for Efficient Spectral Rendering

TL;DR

A lightweight neural upsampling network is introduced that maps RGB assets directly to latent codes, enabling integration of legacy RGB content into the spectral pipeline while maintaining perceptually accurate colors in rendered images.

Abstract

Spectral rendering accurately reproduces wavelength-dependent appearance but is computationally expensive, as shading must be evaluated at many wavelength samples and scales roughly linearly with the number of samples. It also requires spectral textures and lights throughout the rendering pipeline. We propose Hadamard spectral codes, a compact latent representation that enables spectral rendering using standard RGB rendering operations. Spectral images are approximated with a small number of RGB rendering passes, followed by a decoding step. Our key requirement is latent linearity: scaling and addition in spectral space correspond to scaling and addition of codes, and the element-wise product of spectra (for example reflectance times illumination) is approximated by the element-wise product of their latent codes. We show that an exact low-dimensional algebra-preserving representation cannot exist for arbitrary spectra when the latent dimension k is smaller than the number of spectral samples n. We therefore introduce a learned non-negative linear encoder and decoder architecture that preserves scaling and addition exactly while encouraging approximate multiplicativity under the Hadamard product. With k = 6, we render k/3 = 2 RGB images per frame using an unmodified RGB renderer, reconstruct the latent image, and decode to high-resolution spectra or XYZ or RGB. Experiments on 3D scenes demonstrate that k = 6 significantly reduces color error compared to RGB baselines while being substantially faster than naive n-sample spectral rendering. Using k = 9 provides higher-quality reference results. We further introduce a lightweight neural upsampling network that maps RGB assets directly to latent codes, enabling integration of legacy RGB content into the spectral pipeline while maintaining perceptually accurate colors in rendered images.

Figures (8)

• Figure 2: Overview of our learned Hadamard spectral codec and multi-pass rendering pipeline. (a) Training: The encoder $\mathcal{E}$ (with weights $W_\text{enc}$) maps reflectance $R$ and illumination $L$ spectra to $k$-dimensional latent codes, which are split into $B = k/3$ blocks. These codes undergo a blockwise Hadamard product $\odot_B$, and the decoder $\mathcal{D}$ (with weights $W_\text{dec}$) reconstructs the spectral product $R \odot L$. The multi-objective loss $\mathcal{L}_\text{total}$ enforces reconstruction fidelity and near-multiplicativity. (b) Multi-Pass Rendering Pipeline: At inference time, spectral assets are encoded once into $k$-dimensional codes. Each code is partitioned into $B$ RGB-like triplets, which are rendered independently using an unmodified RGB renderer across $B$ passes. The outputs are reassembled into a latent image and decoded once to produce the final spectral or XYZ/RGB image. (c) Codec Architecture Detail: The encoder and decoder are linear transformations with non-negative weights. The blockwise product operation $\odot_B$ performs element-wise multiplication within each 3-channel block, enabling $B$ independent rendering passes that approximate the spectral product in compressed space.
• Figure 3: Comparison of rendering methods on textured objects: (a) standard RGB rendering, (b) full spectral ground truth, (c) our $k{=}6$ latent rendering, and (d) RGB textures and lights upsampled to $k{=}6$ codes. Our native latent rendering (c) matches ground truth (b), while upsampled RGB assets (d) achieve significantly better color accuracy than RGB rendering (a), as visible in the banana and blue sample holder. This demonstrates integration of RGB assets via latent upsampling, so that legacy RGB content can benefit from spectral rendering accuracy without requiring spectral source data.
• Figure 4: Cornell box scene with diverse materials and broadband illumination. (a) Ground truth spectral rendering showing a color checker chart, diffuse walls, and objects with varied BRDFs. (b) Two broadband illuminant SPDs used in the scene (warm and cool). (c) Our native $k{=}6$ latent rendering produces results visually indistinguishable from ground truth. (d) RGB assets (materials, textures, lights) upsampled to $k{=}6$ latent codes and rendered through our pipeline also achieve visually accurate color reproduction.
• Figure 5: Cornell box scene with diverse materials under broadband illumination. (a) Ground truth spectral rendering showing a color checker chart, diffuse walls, and objects with varied BRDFs. (b) Broadband illuminant SPDs used in the scene. (c) Our native $k{=}6$ latent rendering with (d) corresponding per-pixel MSE error map. (e) Our native $k{=}9$ latent rendering with (f) corresponding per-pixel MSE error map. While $k{=}9$ produces lower reconstruction error (mean MSE: [value]), the more practical $k{=}6$ configuration still achieves perceptually accurate results suitable for production rendering.
• Figure 6: Glass material rendering with multi-layer light transport. The scene features a glass outer shell containing colored internal objects (object by Jonas Pilo). Top row: Spectral input data—(a) narrowband SPDs, (b) reflectances of plane and inner object, (c) broadband SPDs. Middle row: Our $k{=}6$ results under (d) narrowband and (e) broadband illumination. Bottom row: Ground truth renderings (f--g). Despite the complex light paths through refractive glass and multiple material interactions, our method achieves visually accurate results under both challenging narrowband and smooth broadband illumination conditions.