SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation

TL;DR

SpecGen tackles the problem of generating spectral BRDFs from a single RGB sphere image by introducing the Spectral-Spatial Tri-plane Aggregation (SSTA), which decouples spatial (angle-based) and spectral (wavelength) information into two tri-planes. An Adaptive Feature Fusion (AFF) module dynamically combines features from both tri-planes, and a lightweight MLP maps fused features to spectral BRDF values. The method uses a RGB–spectral joint training strategy to leverage abundant RGB BRDF data, significantly improving spectral BRDF reconstruction and enabling accurate multispectral rendering under diverse lighting and shapes. Experiments on public BRDF datasets show substantial PSNR gains over hyperspectral baselines (averaging ~8 dB), demonstrating strong generalization and practical relevance for photorealistic spectral rendering.

Abstract

Synthesizing spectral images across different wavelengths is essential for photorealistic rendering. Unlike conventional spectral uplifting methods that convert RGB images into spectral ones, we introduce SpecGen, a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere. This enables spectral image rendering under arbitrary illuminations and shapes covered by the corresponding material. A key challenge in spectral BRDF generation is the scarcity of measured spectral BRDF data. To address this, we propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions, allowing the training strategy to leverage abundant RGB BRDF data to enhance spectral BRDF generation. Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction, achieving an improvement of 8 dB in PSNR. Codes and data will be released upon acceptance.

Figures (4)

• Figure 1: From a single RGB image of a sphere (left), our method recovers the spectral BRDF of the corresponding material, allowing the spectral image rendering of arbitrary illuminations and shapes covered by the material under different wavelengths (right).
• Figure 2: Pipeline of our SpecGen.
• Figure 3: Qualitative comparisons of spectral rendering, where our method takes the RGB sphere input image, generates the spectral BRDF, and renders the spectral bunny image. HSI baselines take the RGB bunny image and directly output the corresponding spectral image.
• Figure 4: Qualitative comparisons of rendering results using only spectral BRDF dataset RGL-S vs. spectral plus RGB BRDF dataset RGL-C-S+MERL, where top and bottom parts denote two different test materials. With the help of our training strategy building upon our SSTA network, we can leverage RGB BRDF data to boost the spectral rendering quality.