Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views

TL;DR

The paper addresses the ill-posed problem of hyperspectral reconstruction from RGB by leveraging a low-cost, multi-view smartphone setup that uses external spectral filters on auxiliary cameras to create a 9-channel input. It introduces the MI-HSR framework and the Doomer dataset, and proposes MI-MSFN, an alignment-aware transformer-based network that fuses misaligned multi-view inputs after spatial alignment is learned. Through experiments on real-world data, the approach demonstrates consistent improvements over single-view methods, achieving noticeable gains in PSNR, SAM, and NSE, and shows robustness to misalignment and sensor noise. The work offers a practical route to deployable hyperspectral imaging with consumer hardware, and provides a benchmark for future development in multi-view spectral imaging and reconstruction under real-world conditions.

Abstract

Hyperspectral reconstruction (HSR) from RGB images is a fundamentally ill-posed problem due to severe spectral information loss. Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our configuration, grounded in theoretical and empirical analysis, enables richer and more diverse spectral observations than conventional single-camera setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We show that the proposed HSR model achieves consistent improvements over existing methods on the newly proposed benchmark. In a nutshell, our setup allows 30% towards more accurately estimated spectra compared to an ordinary RGB camera. Our findings suggest that multi-view spectral filtering with commodity hardware can unlock more accurate and practical hyperspectral imaging solutions.

Figures (14)

• Figure 1: Proposed low-cost mobile spectral imaging system that transforms a standard smartphone into a spectrally diverse capture device via external filters on auxiliary cameras. This configuration enables simultaneous, multi-channel acquisition without internal hardware modification, supporting practical and scalable hyperspectral reconstruction.
• Figure 2: Filters selected for Tele and Wide cameras.
• Figure 3: Sample scenes from the Doomer dataset. Smartphone images are rendered to sRGB using device-specific color matrices; hyperspectral images are rendered using CIE RGB CMF.
• Figure 4: Capture setup for the Doomer dataset. (a) Smartphone holder rotated to allow hyperspectral capture via Specim IQ. (b) Smartphone repositioned for simultaneous multi-camera RGB capture. (c) External spectral filters mounted on Tele and Wide cameras to induce spectral diversity.
• Figure 5: Spatial preprocessing pipeline. Left: geometric alignment of RGB views using SIFT + RANSAC for consistent cross-camera registration. Right: field-of-view normalization and resolution matching across RGB and hyperspectral modalities.