Learnable Quantum Efficiency Filters for Urban Hyperspectral Segmentation

Abstract

Hyperspectral sensing provides rich spectral information for scene understanding in urban driving, but its high dimensionality poses challenges for interpretation and efficient learning. We introduce Learnable Quantum Efficiency (LQE), a physics-inspired, interpretable dimensionality reduction (DR) method that parameterizes smooth high-order spectral response functions that emulate plausible sensor quantum efficiency curves. Unlike conventional methods or unconstrained learnable layers, LQE enforces physically motivated constraints, including a single dominant peak, smooth responses, and bounded bandwidth. This formulation yields a compact spectral representation that preserves discriminative information while remaining fully differentiable and end-to-end trainable within semantic segmentation models (SSMs). We conduct systematic evaluations across three publicly available multi-class hyperspectral urban driving datasets, comparing LQE against six conventional and seven learnable baseline DR methods across six SSMs. Averaged across all SSMs and configurations, LQE achieves the highest average mIoU, improving over conventional methods by 2.45\%, 0.45\%, and 1.04\%, and over learnable methods by 1.18\%, 1.56\%, and 0.81\% on HyKo, HSI-Drive, and Hyperspectral City, respectively. LQE maintains strong parameter efficiency (12--36 parameters compared to 51--22K for competing learnable approaches) and competitive inference latency. Ablation studies show that low-order configurations are optimal, while the learned spectral filters converge to dataset-intrinsic wavelength patterns. These results demonstrate that physics-informed spectral learning can improve both performance and interpretability, providing a principled bridge between hyperspectral perception and data-driven multispectral sensor design for automotive vision systems.

Figures (5)

• Figure 1: Comparison of visual images, RGB Planes, and hypercube.
• Figure 2: Overview of conventional imaging pipelines and the proposed learnable quantum efficiency (LQE) framework. (Top) Standard RGB systems employ fixed Bayer CFA, producing low-dimensional data that are computationally efficient but susceptible to metamerism and loss of material-specific information. (Middle) HSI captures dense spectra but incurs substantial computational and memory overhead due to high dimensionality, requiring DR. (Bottom) The proposed LQE approach introduces continuously differentiable, high-order QE filters that are jointly optimized with the downstream perception model and optically plausible. This framework learns a compact, task-adaptive spectral projection of the hypercube, mitigating the computational burden of HSI while preserving discriminative spectral content.
• Figure 3: Example QE curves of the ZWO ASI662MC (left) and ASI462MC (right) sensors' CFA ZWOAstro_ASI662MC_ASI462MC, exhibiting a dominant spectral peak, smooth spectral response, and bounded bandwidth.
• Figure 4: Comparison of learnable DR layers. All methods transform input hypercubes from $C$ spectral channels to $F$ reduced channels ($F \ll C$) while preserving spatial dimensions $H \times W$. (a) 1×1Conv: Standard Conv with batch normalization and ReLU. (b) AE. (c) seAttn: Gating mechanism with SE channel reweighting. (d) DSC: Depthwise separable convolution. (e) CN: ConvNeXt-style block. Others include original CBAM and ECA style layers. (f) Proposed LQE method parameterizes spectral response filters as high-order Gaussian basis functions $Q_f$. Numbers in parentheses indicate: kernel sizes. BN: Batch Normalization, and LN: Layer Normalization.
• Figure 5: LQE spectral response filters on HyKo-VIS dataset with $F=3$ and $P=3$. Filters are trained on different backbones and initialized with random perturbations as in Epoch-1. Both architectures converge to similar spectral response patterns by the final epoch, as evidenced by the substantial shift in Filter 1 (blue curves) from the blue-green region ($\sim$510 nm) toward the blue region ($\sim$490 nm). Solid curves represent the complete LQE filter responses, while dashed vertical lines indicate starting peak centroids (blue/green). Red dashed lines highlight spectral regions where filter coverage transitions, showing how two adjacent filters adapt to maintain complete spectral coverage as one filter shifts, specifically at $\sim$530nm.