Single-sample image-fusion upsampling of fluorescence lifetime images

TL;DR

This work tackles the resolution-speed trade-off in fluorescence lifetime imaging (FLIM) by introducing SiSIFUS, a data-fusion framework that reconstructs a high-resolution lifetime map $\tau_{HR}$ from a sparse lifetime measurement $\tau_{LR}$ and a high-resolution intensity image $I$. It leverages two priors—local lifetime–intensity mappings and morphology-driven global predictions via a neural network—within a TV-regularized ADMM inverse-retrieval, without relying on external training data. Across multiple biological samples and imaging modalities, SiSIFUS yields sharper, more perceptually faithful lifetime maps than bilinear interpolation and offers substantial acquisition-time reductions in scanning setups, while remaining robust to noise and avoiding overfitting to non-existent global features. The approach is generalizable to other two-modality SR problems and provides a practical path toward faster, high-resolution FLIM with minimal hardware changes.

Abstract

Fluorescence lifetime imaging microscopy (FLIM) provides detailed information about molecular interactions and biological processes. A major bottleneck for FLIM is image resolution at high acquisition speeds, due to the engineering and signal-processing limitations of time-resolved imaging technology. Here we present single-sample image-fusion upsampling (SiSIFUS), a data-fusion approach to computational FLIM super-resolution that combines measurements from a low-resolution time-resolved detector (that measures photon arrival time) and a high-resolution camera (that measures intensity only). To solve this otherwise ill-posed inverse retrieval problem, we introduce statistically informed priors that encode local and global dependencies between the two single-sample measurements. This bypasses the risk of out-of-distribution hallucination as in traditional data-driven approaches and delivers enhanced images compared for example to standard bilinear interpolation. The general approach laid out by SiSIFUS can be applied to other image super-resolution problems where two different datasets are available.

Figures (11)

• Figure 1: Schematic of the local prior method.(A) Shown are a CMOS (fluoresence intensity) field of view, with the SPAD field of view (fluorescence lifetime), overlayed on top of it so as to match the sparse, low fill-factor pixel layout of the SPAD array. (B) We zoom in on a 5 × 5 window. All SPAD pixels have a corresponding CMOS measurement, but so do the areas in-between SPAD pixels. We aim to find the lifetime at points with no SPAD samples. For this, we fit a function, for instance linear interpolation, a cubic spline or a radial basis function gaussian process. Then, the high-resolution CMOS pixels $x_{HR}$ which we wish to upsample are fitted with this function, producing a lifetime estimate $\hat{tau}_{HR}$. (C) We slide the window across the field of view, fitting new functions for each new window, and predicting the centres, upsampling the FLIM image to the resolution of the intensity image, window-by-window.
• Figure 1: We found the mean LPIPS for priors generated using various window sizes, averaged across our 4 samples and 4 upsampling factors (2,4,8,16). We plot both the geometric and arithmetic mean.
• Figure 2: Schematic of the global prior method.(A) Fluorescence intensity of a convallaria - acridine orange sample, with 8 × 8 sparse lifetime samples overlayed. We extract intensity patches from this image; a few of them correspond to a central lifetime sample. Such patches are training data, which we can use to predict the central lifetime of the rest of the patches. (B) Training inputs (patches) are augmented via rotation and mirroring. They can be further augmented by adding the patches which are nearest neighbours of training patches and allocating them the same label (lifetime) as the sampled patch. The deep neural network (DNN) architecture is simple, consisting of three 2D convolutional layers followed by three fully connected layers. (C) Finally, the trained DNN evaluates patches with unsampled centres, thus super-resolving the lifetime image.
• Figure 2: We found the mean MAE and LPIPS for priors generated using various LCP types, averaged across our 4 samples and 4 upsampling factors (2,4,8,16).
• Figure 3: 16x16 upsampling of MDCK cells.(A) Low resolution fluorescence lifetime image (32x32) of Madin-Darby canine kidney (MDCK) cells expressing Flipper-TR dye. (B) Corresponding high resolution intensity image (512x512) of the sample. (C) 5x5 windows of low-resolution FLIM are fitted to corresponding intensity values, to generate a local prior image (two example windows are shown). (D) A global prior image is generated from 13x13 intensity patches with central FLIM measurements (two examples are shown). (E) The ground truth high-resolution FLIM target, intensity weighted for visualisation. (F) The proposed method, upsampling the low-resolution measurement by a factor of 16x16. (G) Bilinear interpolation upsampling the FLIM measurement by 16x16.