HyperGS: Hyperspectral 3D Gaussian Splatting

TL;DR

HyperGS tackles hyperspectral novel view synthesis by embedding hyperspectral data into a compact latent space and performing Gaussian splatting in that space. It couples a fast latent-space autoencoder with a NeRF-style MLP to model view-dependent spectral effects per Gaussian, while introducing adaptive density densification and pixel-wise pruning to maintain spectral fidelity and efficiency. The approach achieves state-of-the-art performance on real and simulated hyperspectral datasets, with up to around 14 dB PSNR gains over prior HNVS methods and strong robustness to varying spectral cameras. By providing a new HNVS benchmark and detailed ablations, the work demonstrates scalable, material-aware spectral rendering suitable for real-time-like applications.

Abstract

We introduce HyperGS, a novel framework for Hyperspectral Novel View Synthesis (HNVS), based on a new latent 3D Gaussian Splatting (3DGS) technique. Our approach enables simultaneous spatial and spectral renderings by encoding material properties from multi-view 3D hyperspectral datasets. HyperGS reconstructs high-fidelity views from arbitrary perspectives with improved accuracy and speed, outperforming currently existing methods. To address the challenges of high-dimensional data, we perform view synthesis in a learned latent space, incorporating a pixel-wise adaptive density function and a pruning technique for increased training stability and efficiency. Additionally, we introduce the first HNVS benchmark, implementing a number of new baselines based on recent SOTA RGB-NVS techniques, alongside the small number of prior works on HNVS. We demonstrate HyperGS's robustness through extensive evaluation of real and simulated hyperspectral scenes with a 14db accuracy improvement upon previously published models.

Figures (8)

• Figure 1: Visual system diagram of our approach. Blue lines indicate the operational flow, while orange lines represent the gradient flow. Our novel modified latent hyperspectral adaptive density function operates within a latent space provided by a frozen autoencoder, which is also responsible for decoding the final images. Latent space novel views are generated through a combination of the latent hyperspectral Gaussian point cloud and a NeRF-style MLP. Gradients are updated based on the decoded images.
• Figure 2: Our channel-wise convolutional AE model learns LHSI space representation of the scene. The decoder is only used in training and inference for the 3DGS system after the preprocessing of the dataset is finished.
• Figure 3: Visualization of our re-projection protocol for initializing the SFM point cloud. We estimate the SFM from grayscale channel slices of the hyperspectral image scene with COLMAP. Then, using the average of all views, we re-project each point into LHSI from provided by our AE, providing an optimal initialization of latent spectral color for the 3DGS system.
• Figure 4: Visualization of our pixel-wise pruning. Gaussians with poor similarity scores below a threshold are pruned, such as the red-circled Gaussian and the black ground truth spectrum.
• Figure 5: Visualisation of the top 4 methods for frame 51 of 359 for the Caladium plant scene from the Bayspec dataset. The top row shows the 70th channel of the 141 channel image predicted, the bottom row provides a raw pixel-wise error heatmap of the scene.