CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis

TL;DR

CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities, is introduced, featuring a novel spectral encoder, featuring a self-attention-cross-attention mechanism, to distill salient spectral information into learned spectral representations.

Abstract

Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding, and is already established as a critical modality in remote sensing. However, variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies, leading to camera-specific models with limited generalizability and inadequate cross-camera applicability. To address this bottleneck, we introduce CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities. To enable the conversion of a spectral image with any channel dimensionality to a camera-agnostic representation, we introduce a novel spectral encoder, featuring a self-attention-cross-attention mechanism, to distill salient spectral information into learned spectral representations. Spatio-spectral pre-training is achieved with a novel feature-based self-supervision strategy tailored to CARL. Large-scale experiments across the domains of medical imaging, autonomous driving, and satellite imaging demonstrate our model's unique robustness to spectral heterogeneity, outperforming on datasets with simulated and real-world cross-camera spectral variations. The scalability and versatility of the proposed approach position our model as a backbone for future spectral foundation models. Code and model weights are publicly available at https://github.com/IMSY-DKFZ/CARL.

Figures (6)

• Figure 1: CARL addresses spectral camera variations by learning camera-agnostic representations. Unlike existing methods that require retraining for each channel configuration, CARL generalizes across cameras and outperforms both camera-specific and channel-invariant approaches across domains. The model processes one image at a time, ensuring flexibility without dependence on fusion strategies.
• Figure 2: Conversion of a camera-specific spectral image into a camera-agnostic representation. To address the heterogeneity in camera-dependent spectral properties, a dedicated spectral encoder extracts a camera-agnostic representation by leveraging spectral tokens encoding wavelength information. A spectral image of dimension $H\times W\times C$ is divided into patches of size $P$ and projected band-wise into a $D$-dimensional feature space. The spectral encoder $E_{\text{spec}}$ processes each patch individually, and hereby resolves the spectral dimension. In particular, $E_{\text{spec}}$ encodes the wavelength $\lambda_i$ of channel $i$ as positional encoding $PE(\lambda_i)$ and adds it to the embedded patch $\Lambda_i$. Self-Attention across spectral tokens $(\Lambda_i)_{i\leq C}$ and Cross-Attention with $K$ learned spectral representations yield enriched representations $(S_j)_{j\leq K}$. After aggregation into a camera-agnostic representation, a standard image encoder, $E_{\text{spat}}$, captures spatial relationships.
• Figure 3: CARL-SSL enables joint learning of camera-agnostic representations and spatial relations. Spectral self-supervision involves reconstruction of masked spectral channels in feature space. The student $E_{\text{spec}}$ extracts spectral representations $(S_j)_{j\leq K}$ from a spectrally masked input, while the predictor $\phi_{\text{spec}}$ predicts the masked spectral tokens using masked wavelengths , and $(S_j)_{j\leq K}$. Target tokens are generated by the teacher $\Tilde{E}_{\text{spec}}$ from the complete input. The aggregated camera-agnostic representations are subsequently processed by I-JEPA.
• Figure 4: Our model enables cross-modality knowledge transfer.(Left) With increasing spectral heterogeneity in the training set, both the camera-specific model and Hyve exhibit a notable rise in prediction noise. In contrast, CARL consistently provides accurate predictions. (Right) In two HSICity test set examples, the HSI-specific model fails to segment "poles" (gray labels) due to their absence in the HSICity training set. CARL, however, jointly trained on RGB and HSI data, effectively leverages "pole" annotations from Cityscapes to inform its predictions on HSICity.
• Figure 5: Our model shows unique robustness to spectral heterogeneity in the organ experiments. As spectral heterogeneity increases with the multispectral replacements in the training set, CARL uniquely maintains a high mIoU score on the hyperspectral test set. Shaded area: 95% confidence interval.