Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

Andrew Wang, Mike Davies

Abstract

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On intraoperative and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance.

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

Abstract

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On intraoperative and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance.
Paper Structure (19 sections, 3 equations, 7 figures, 2 tables)

This paper contains 19 sections, 3 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-supervised demosaicing
  4. Fine-tuning without ground truth
  5. Methodology
  6. Projective geometry in camera systems
  7. Perspective-equivariant fine-tuning for demosaicing
  8. Extension to joint demosaicing and denoising
  9. Experiments & Analysis
  10. Experimental setup
  11. Data
  12. Model
  13. Evaluation
  14. Baselines
  15. Results
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Perspective-Equivariant Fine-tuning for Demosaicing (PEFD) recovers sharp multispectral images with correct colours without ground truth, compared to popular classical methods such as weighted bilinear demosaicing or learning from scratch.
  • Figure 2: Given a camera centre $C$, camera systems in intraoperative (left) and automotive (right) imaging rotate freely about their axes, producing images related by perspective transformations.
  • Figure 3: Framework for perspective-equivariance fine-tuning for multispectral demosaicing, where ${f_\theta}$ is our adapted pretrained foundation model terris_reconstruct_2025 to process multispectral data, ${\mathbf y}$ are the mosaiced measurements, ${\mathbf A}$ is the mosaicing operator, ${\mathbf{T}_g}$ is the parametrised perspective transformation and the loss depicts \ref{['eq:ei_loss']}.
  • Figure 4: Test set false-RGB reconstruction results on 3 example neurosurgical intraoperative images from HELICoiD fabelo_-vivo_2019, showing mosaiced input ${\mathbf y}$, reconstruction results for 9 comparison methods, our reconstruction with PEFD, and ground truth for reference.
  • Figure 5: Example spectral signatures from a sample image patch in each dataset (left: HELICoiD; right: HyKo).
  • ...and 2 more figures