scSplit: Bringing Severity Cognizance to Image Decomposition in Fluorescence Microscopy

TL;DR

scSplit addresses the problem of image decomposition in fluorescence microscopy when the relative intensities of superimposed structures vary and bleeding through between channels occurs. It introduces Severity Cognizant Input Normalization (SCIN) to normalize inputs across all mixing severities and a regressor to estimate the mixing ratio, guided by an InDI-inspired degradation model with two dedicated generative networks Gen$_0$ and Gen$_1$. The method is trained with a mixing-ratio distribution that emphasizes mid-severity and uses aggregation across acquisition sessions to stabilize estimates, enabling simultaneous image unmixing and bleedthrough removal. Empirically, scSplit achieves state-of-the-art or competitive PSNR and MS-SSIM across five real datasets and shows improved downstream segmentation performance, while ablations confirm the value of SCIN, the Reg network, and the aggregation strategy for robustness to mixing-ratio variations.

Abstract

Fluorescence microscopy, while being a key driver for progress in the life sciences, is also subject to technical limitations. To overcome them, computational multiplexing techniques have recently been proposed, which allow multiple cellular structures to be captured in a single image and later be unmixed. Existing image decomposition methods are trained on a set of superimposed input images and the respective unmixed target images. It is critical to note that the relative strength (mixing ratio) of the superimposed images for a given input is a priori unknown. However, existing methods are trained on a fixed intensity ratio of superimposed inputs, making them not cognizant of the range of relative intensities that can occur in fluorescence microscopy. In this work, we propose a novel method called scSplit that is cognizant of the severity of the above-mentioned mixing ratio. Our idea is based on InDI , a popular iterative method for image restoration, and an ideal starting point to embrace the unknown mixing ratio in any given input. We introduce (i) a suitably trained regressor network that predicts the degradation level (mixing ratio) of a given input image and (ii) a degradation-specific normalization module, enabling degradation-aware inference across all mixing ratios. We show that this method solves two relevant tasks in fluorescence microscopy, namely image splitting and bleedthrough removal, and empirically demonstrate the applicability of scSplit on 5 public datasets. The source code with pre-trained models is hosted at https://github.com/juglab/scSplit/.

Figures (25)

• Figure 1: Schematic overview of the $\text{sc}\mathbb{S}\text{plit}$ framework for handling image superposition at varying severity levels.(Left) Training pipeline: The input to the system is a superimposed image, generated as a weighted average of two images using a mixing ratio $t \in [0, 1]$. The superimposed image is passed through a normalization module, which performs ratio-specific normalization to ensure zero mean and unit standard deviation. The normalized image is then processed by two generative networks, $\textit{Gen}_0$ and $\textit{Gen}_1$, to estimate the individual structures. A regressor network, Reg, is trained to predict the mixing ratio $t$ from a normalized superimposed image. (Right) Inference pipeline: During inference, the mixing ratio $t$ is estimated for a set of superimposed input images using Reg, and the estimates are aggregated to obtain $t_{\text{agg}}$. The normalized superimposed images, along with $t_{\text{agg}}$, are fed into the generative networks $\textit{Gen}_i$ to recover the individual structures. Thanks to the mixing-ratio specific normalization during training, the normalization during inference is simple and is performed using the mean and standard deviation computed from the set of test input patches.
• Figure 2: Qualitative evaluation of unmixing performance. We show qualitative evaluation on Hagen et al. Hagen2021-xh (top panel) and BioSR biosrdataset (bottom panel). For each dataset, the full input frame (top-left) and a zoomed-in input patch (bottom-left) are displayed. Predictions and corresponding targets (last column) are for the input patch. They are shown for both channels, with each channel displayed in a separate row. PSNR values are also reported for the predicted patch. The mixing ratio $w$ indicated above the input column corresponds to the first channel, with the second channel naturally having a ratio of ($1-w$). Additional qualitative evaluations across different $w$ values for all datasets are provided in the Supplementary Figures 4 through 18.
• Figure 3: Performance evaluation of the regressor network Reg . (a) Quantitative evaluation of performance degradation with incorrect $w$ during inference: This analysis highlights the sensitivity to inaccurate predictions by Reg . The x-axis represents the assumed $w$ during inference, the legend represents the actual $w$, and the y-axis quantifies the performance. (b) Qualitative evaluation on the BioSR dataset with varying $w$ during inference: Predictions are shown for each channel (two rows) under different assumed $w$ values. For each $w$, the input is divided into upper and lower halves, displayed in two sub-rows. The first sub-row sets negative pixel values to zero during visualization, while the second sub-row uses default visualization. It is worth noting that significant artefacts (tiling artifacts, disappearance of structures, and increased "crispness" of microtubule curves) get manifested when the assumed $w$ is reasonably far from $w=0.8$, the $w$ value used to create the input.
• Figure 4: Analysis of factors contributing to the superior performance of $\text{sc}\mathbb{S}\text{plit}$. (Left) Justification for the normalization module on the Hagen et al. dataset: The conventional normalization of $C_0$ and $C_1$ leads to out-of-distribution issue during test evaluation. In contrast, our $w$-specific normalization scheme demonstrates superior performance. (Right) Comparison against suitable augmentations: We investigate one key factor behind $\text{sc}\mathbb{S}\text{plit}$'s enhanced performance: its exposure to inputs with varying levels of mixing during training. For this, we introduce two augmentations to the input generation process of $\mu\text{Split}_D$Ashesh2023-wtf, one of our baselines, allowing it to also observe different mixing levels during training. While these augmented $\mu\text{Split}_D$ variants show improved performance over the vanilla $\mu\text{Split}_D$, $\text{sc}\mathbb{S}\text{plit}$ consistently outperforms them across mixing ratios $w$.
• Figure S.1: Handling varying levels of superposition. For the objective of image unmixing task, superimposed images acquired with Fluorescence microscopy can have varying levels of mixing of the constituent structures. Additionally, insufficiently precise optical filtering often leads to 'bleedthrough' wherein a structure of interest gets superimposed with a shadowed presence of another structure. $\text{sc}\mathbb{S}\text{plit}$ uniquely addresses these varying levels of structural mixing in the superimposed input images. Unlike existing unmixing methods, $\text{sc}\mathbb{S}\text{plit}$'s architecture adapts to different degrees of superposition and accounts for the resulting variations in pixel intensity distributions (inset plot in red), enabling effective input image normalization and leading to efficient unmixing across diverse mixing ratios.