Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging

Peichao Li, Oscar MacCormac, Jonathan Shapey, Tom Vercauteren

TL;DR

This paper tackles real-time hyperspectral demosaicking for snapshot mosaic cameras in neurosurgical imaging without relying on paired high-resolution hyperspectral ground truth. It presents a self-supervised, cycle-consistent adversarial framework that uses unpaired RGB images from surgical microscopes to guide spectral reconstruction, paired with a trainable RGB converter and a spectral recovery network. A novel inverse pixel shuffle loss reduces gridding artifacts, and a data-fidelity override ensures accurate reconstruction. The method demonstrates superior quantitative and qualitative results, real-time performance, and favorable clinician feedback, indicating strong potential for integration into intra-operative workflows.

Abstract

Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. However, snapshot mosaic imaging requires a demosaicking algorithm to fully restore the spatial and spectral details in the images. Modern demosaicking approaches typically rely on synthetic datasets to develop supervised learning methods, as it is practically impossible to simultaneously capture both snapshot and high-resolution spectral images of the exact same surgical scene. In this work, we present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth. We leverage unpaired standard high-resolution surgical microscopy images, which only provide RGB data but can be collected during routine surgeries. Adversarial learning complemented by self-supervised approaches are used to drive our hyperspectral-based RGB reconstruction into resembling surgical microscopy images and increasing the spatial resolution of our demosaicking. The spatial and spectral fidelity of the reconstructed hyperspectral images have been evaluated quantitatively. Moreover, a user study was conducted to evaluate the RGB visualisation generated from these spectral images. Both spatial detail and colour accuracy were assessed by neurosurgical experts. Our proposed self-supervised demosaicking method demonstrates improved results compared to existing methods, demonstrating its potential for seamless integration into intra-operative workflows.

A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging

TL;DR

This paper tackles real-time hyperspectral demosaicking for snapshot mosaic cameras in neurosurgical imaging without relying on paired high-resolution hyperspectral ground truth. It presents a self-supervised, cycle-consistent adversarial framework that uses unpaired RGB images from surgical microscopes to guide spectral reconstruction, paired with a trainable RGB converter and a spectral recovery network. A novel inverse pixel shuffle loss reduces gridding artifacts, and a data-fidelity override ensures accurate reconstruction. The method demonstrates superior quantitative and qualitative results, real-time performance, and favorable clinician feedback, indicating strong potential for integration into intra-operative workflows.

Abstract

Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. However, snapshot mosaic imaging requires a demosaicking algorithm to fully restore the spatial and spectral details in the images. Modern demosaicking approaches typically rely on synthetic datasets to develop supervised learning methods, as it is practically impossible to simultaneously capture both snapshot and high-resolution spectral images of the exact same surgical scene. In this work, we present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth. We leverage unpaired standard high-resolution surgical microscopy images, which only provide RGB data but can be collected during routine surgeries. Adversarial learning complemented by self-supervised approaches are used to drive our hyperspectral-based RGB reconstruction into resembling surgical microscopy images and increasing the spatial resolution of our demosaicking. The spatial and spectral fidelity of the reconstructed hyperspectral images have been evaluated quantitatively. Moreover, a user study was conducted to evaluate the RGB visualisation generated from these spectral images. Both spatial detail and colour accuracy were assessed by neurosurgical experts. Our proposed self-supervised demosaicking method demonstrates improved results compared to existing methods, demonstrating its potential for seamless integration into intra-operative workflows.
Paper Structure (11 sections, 4 equations, 4 figures, 2 tables)

This paper contains 11 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Cycle-consistent adversarial training
  4. Converting Between Hyperspectral and RGB Images
  5. Inverse Pixel Shuffle loss
  6. Experiments and Results
  7. Surgical imaging data
  8. Training details
  9. Quantitative results
  10. User study
  11. Conclusion

Figures (4)

  • Figure 1: Diagram of the proposed demosaicking algorithm. First, bilinear interpolation is applied to the input snapshot mosaic image to recover a fully-sampled spatial and spectral grid. The interpolated image then serves as input to the demosaicking network $G_{\text{demos}}$, which generates the refined hyperspectral image. Super-resolution losses and adversarial losses are computed for training the network.
  • Figure 2: (a) MLP model for hyperspectral-to-RGB conversion. (b) An example of periodic gridding artefacts that can be observed with state-of-the-art hyperspectral demosaicking li2023spatial. (c) Diagram illustrating the Inverse Pixel Shuffle (IPS) operation.
  • Figure 3: Comparison between different demosaicking methods on an example NeuroHSI test image.
  • Figure 4: Box plot illustrating the difference between our demosaicking results and linear demosaicking. Differences are expected whenever edges occur in the images as linear demosaicking otherwise results in low-resolution reconstruction. This analysis shows the absence of spectral bias in our reconstructions.