Hyperspectral Image Restoration and Super-resolution with Physics-Aware Deep Learning for Biomedical Applications

TL;DR

This work tackles the trade-off between spatial, spectral resolution, and imaging speed in hyperspectral and mass spectrometry imaging by introducing HyReS, a physics-aware DL-SISR framework that restores and upscales images post-acquisition without requiring extra training data. By incorporating a Fourier-domain loss (FRC) into a GAN-based restoration model (FRCGAN), HyReS delivers substantial gains in resolution (up to $16\times$) and speed ($12\times$), while preserving biological fidelity across synthetic and diverse experimental datasets. The approach yields improved downstream performance in segmentation and classification tasks (e.g., MS lesion detection and Down syndrome metabolic phenotypes) and provides interpretable insights into the imaging process via metrics like the difference PSF and cross-modal concordance with IMC data. Overall, HyReS offers a practical, explainable pathway to approach hardware-imposed limits in bioimaging and enables high-throughput, high-resolution spatial omics analyses; the authors release open-source software to facilitate adoption and further development.

Abstract

Hyperspectral imaging is a powerful bioimaging tool which can uncover novel insights, thanks to its sensitivity to the intrinsic properties of materials. However, this enhanced contrast comes at the cost of system complexity, constrained by an inherent trade-off between spatial resolution, spectral resolution, and imaging speed. To overcome this limitation, we present a deep learning-based approach that restores and enhances pixel resolution post-acquisition without any a priori knowledge. Fine-tuned using metrics aligned with the imaging model, our physics-aware method achieves a 16X pixel super-resolution enhancement and a 12X imaging speedup without the need of additional training data for transfer learning. Applied to both synthetic and experimental data from five different sample types, we demonstrate that the model preserves biological integrity, ensuring no features are lost or hallucinated. We also concretely demonstrate the model's ability to reveal disease-associated metabolic changes in Downs syndrome that would otherwise remain undetectable. Furthermore, we provide physical insights into the inner workings of the model, paving the way for future refinements that could potentially surpass instrumental limits in an explainable manner. All methods are available as open-source software on GitHub.

Figures (9)

• Figure 1: The 3S space limiting hyperspectral imaging in general and competitor analysis of major approaches granting chemical contrast comparable to that of MSI, with respect to the proposed DL-SISR scheme.
• Figure 2: Restoration of synthetically downsampled breast biopsy images and effect on segmentation. a) Comparison of image segmentation results from both the ground truth (GT) images and those that have been restored by HyReS after downsampling. The optical image of an adjacent tissue that has been H&E-stained is also shown, whose pathological annotations also provided tentative assignment of the segmented regions. Namely, Red = Tumour, Green = Muscle, Blue = Fat. b) Comparison of the clustering and hence image segmentation behaviour for both the GT and HyReS-restored data in a dimensionally reduced 2D domain provided by UMAP. The pathological annotations are illustrated in the same colours as in a).
• Figure 3: a) Comparison of RGB ion images (Red:m/z 150.0 Green:m/z 600.5 Blue:m/z 864.6) of mouse brains that have been acquired at low resolution (LR) with 100µm pixels, at high resolution (HR) with 25µm pixels and when restored to HR from LR by HyReS. A linescan of the pixel intensities along the cerebellum is also shown in each case to depict the relative SNR. b) Image quality evaluation of bicubically downsampled (LR), ground truth high resolution (HR) and HyReS-upsampled images using CRSIQUE scores and estimated resolutions.
• Figure 4: a)H&E stained image from an adjacent slide showing regions-of-interest used of predictive modelling, containing areas of AL/DS/Lesion. b) Predictive modelling results. c) Ion images of key metabolites driving the ground truth classification model before & after HyReS restoration. d) Mean spectra in the spatial metabolomic data before & after HyReS restoration.
• Figure 5: a) ROC curves summarising the predictive modelling performance of trained models generated from GT and HyReS-enhanced images when applied to validation and independent classification tasks. b) Representative overlay images showing the spatial distribution and intensity of regions predicted to be trisomic when applying the HyReS-enhanced model on whole brain images used for training. The DS status and caging arrangement of the mice before imaging are also indicated, where green corresponds to a trisomic transgenic mouse, and white a normal mouse. c) Overlay images illustrating trisomic prediction when applying the HyReS-enhanced model on independent whole brain images. The same DS status and caging arrangement from b) also apply.