Deep intra-operative illumination calibration of hyperspectral cameras

TL;DR

The paper addresses the problem that intraoperative hyperspectral imaging is highly sensitive to dynamic lighting, which disrupts calibration and workflow. It proposes a data-driven, spatially resolved white-reference prediction using a 3D CNN autoencoder trained with a two-dataset paradigm and supported by physics-based illumination simulations for LEDs and halogens. The approach outperforms traditional RGB calibration methods and a prior learning-based method, achieving near-gold-standard spectral calibration and substantial improvements in semantic segmentation and physiological parameter estimation, with demonstrated generalization across species and lighting conditions (e.g., DSC improvements up to 191% and oxygen-saturation errors reduced by 50–69%). This workflow-friendly calibration could become a central component of clinical HSI, enabling robust, real-time use in surgical settings.

Abstract

Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. Currently available cameras, however, suffer from poor integration into the clinical workflow because they require the lights to be switched off, or the camera to be manually recalibrated as soon as lighting conditions change. Given this critical bottleneck, the contribution of this paper is threefold: (1) We demonstrate that dynamically changing lighting conditions in the operating room dramatically affect the performance of HSI applications, namely physiological parameter estimation, and surgical scene segmentation. (2) We propose a novel learning-based approach to automatically recalibrating hyperspectral images during surgery and show that it is sufficiently accurate to replace the tedious process of white reference-based recalibration. (3) Based on a total of 742 HSI cubes from a phantom, porcine models, and rats we show that our recalibration method not only outperforms previously proposed methods, but also generalizes across species, lighting conditions, and image processing tasks. Due to its simple workflow integration as well as high accuracy, speed, and generalization capabilities, our method could evolve as a central component in clinical surgical HSI.

Figures (8)

• Figure 1: Motivation: Current hyperspectral cameras, which require known lighting conditions, fail in real-world scenarios with dynamically changing lighting conditions.
• Figure 2: The proposed approach replaces tedious manual calibration with a dynamic fully-automatic approach. The core of our data-centric method is a 3D-convolutional neural network, trained on in vivo data with artificial light manipulations. At inference time, it takes a raw hyperspectral image as input and generates the corresponding white reference image. The prediction of the white tile image can be used for subsequent calibration of the input image.
• Figure 3: Testing concept based on data from three species.
• Figure 4: State-of-the-art methods fail under dynamically changing light conditions. Our approach addresses this issue. (Left) Results on colorchecker dataset ds_test_cc. The boxplots show the cosine similarity between recalibrated and reference spectra, averaged across colors. Red line: Gold standard of manual white tile calibration. (Right) Results on semantic segmentation dataset ds_test_pig. Red line: Mean DSC in the absence of stray light. Points: Different stray light scenarios.
• Figure 5: In contrast to related methods, our approach generalizes across species. (Left) Organ-specific absolute oxygen saturation errors between calibrated rat images without stray light and corresponding stray light images that are recalibrated by one of the methods. Red line: Mean performance of the gold standard (manual white tile calibration). (Right) Our method yields precise hemoglobin index estimates under dynamically changing lighting conditions.