denoiSplit: a method for joint microscopy image splitting and unsupervised denoising

TL;DR

denoiSplit tackles the problem of joint semantic image splitting and unsupervised denoising in fluorescence microscopy, where imaging noise degrades the ability to separate semantic structures. It introduces a Variational Splitting Encoder-Decoder (VSE) network built on a Hierarchical VAE, augmented with pixel-noise models to jointly denoise and unmix channels, and uses an altered KL loss together with calibrated uncertainty via posterior sampling. Key contributions include the Per-level KL weighting formulation, the integration of pixel-noise models into the generative loss, and a calibration framework that learns per-channel scalars to map posterior variability to predictive error. Across BioSR and Hagen datasets with synthetic and real noise, denoiSplit outperforms baselines such as muSplit and sequential HDN+muSplit, while providing uncertainty estimates that support reliable, end-to-end analysis for multi-structure fluorescence imaging.

Abstract

In this work, we present denoiSplit, a method to tackle a new analysis task, i.e. the challenge of joint semantic image splitting and unsupervised denoising. This dual approach has important applications in fluorescence microscopy, where semantic image splitting has important applications but noise does generally hinder the downstream analysis of image content. Image splitting involves dissecting an image into its distinguishable semantic structures. We show that the current state-of-the-art method for this task struggles in the presence of image noise, inadvertently also distributing the noise across the predicted outputs. The method we present here can deal with image noise by integrating an unsupervised denoising subtask. This integration results in improved semantic image unmixing, even in the presence of notable and realistic levels of imaging noise. A key innovation in denoiSplit is the use of specifically formulated noise models and the suitable adjustment of KL-divergence loss for the high-dimensional hierarchical latent space we are training. We showcase the performance of denoiSplit across multiple tasks on real-world microscopy images. Additionally, we perform qualitative and quantitative evaluations and compare the results to existing benchmarks, demonstrating the effectiveness of using denoiSplit: a single Variational Splitting Encoder-Decoder (VSE) Network using two suitable noise models to jointly perform semantic splitting and denoising.

Figures (20)

• Figure 1: Teaser Figure. In this work we use a variational encoder-decoder network to jointly solve an usupervised denoising and image splitting task and show that our approach outperforms existing baselines.
• Figure 2: Qualitative Results. We show examples of noisy inputs, individual noisy channel training data (GT), and predictions by one of the baselines ($\mu\text{Split}$) and our own results obtained with $\text{denoi}\mathbb{S}\text{plit}$ for four tasks (A: MT vs. CCPs, B: ER vs. CCPs, C: MT vs. ER, and D: F-actin vs. ER). We show high SNR channel images (not used during training) and show PSNR values w.r.t. these images. Additionally, we plot histograms of various panels for comparison (see legend on the right). The bottom cell in the first column of each panel shows the used noise models (see main text for details). The superimposed plots (green) show the distribution of noisy observations ($c_i^N$) for two clean signal intensities.
• Figure 3: Variational Sampling and Calibration. The VSE Network in $\text{denoi}\mathbb{S}\text{plit}$ is capable of sampling from a learned posterior. Here we show cropped inputs ($256\times256$), two corresponding prediction samples, the difference between the two samples ($S_1 - S_2$), the MMSE prediction, and otherwise unused high SNR microscopy for three tasks, namely ER vs. CCPs, ER vs. MT, and CCPs vs. MT. The MMSE predictions are computed by averaging $50$ samples. As before, we show PSNR w.r.t. high SNR patches. The dot plots in the first column show are calibration plots, showcasing that the error estimate we propose works well (see main text).
• Figure 4: Comparison to Sequential Baseline. For each panel (ER vs. CCPs, CCPs vs. MT, and ER vs. MT) we show the full input image and its ($256\times256$) inset crop, corresponding noisy training data crops (GT), the results of the sequential denoising and splitting baseline (HDN$\oplus$$\mu\text{Split}$) and our end-to-end results obtained with $\text{denoi}\mathbb{S}\text{plit}$. All predictions show the MMSE, obtained by averaging $50$ sampled predictions. We show a few zoomed-in locations where the baseline under-performs. Note that such small differences might contribute little to evaluations via PSNR, but can make a huge difference for the downstream analysis of investigated biological structures contained in such microscopy data.
• Figure 5: Results on Actin vs. Mito Task:(Left) Here, qualitative evaluation of the different models on Hagen et al. Hagen2021-xh is shown. We also show High SNR channel images (not used during training) in last column and we show PSNR w.r.t. them. Noise models are shown in column one, second row. (Right) Quantitative evaluation of $\text{denoi}\mathbb{S}\text{plit}$ along with the baselines using PSNR (line 1) and range invariant MS-SSIM Weigert2018-pi (line 2, also see Supp. Sec. 2 for details on the MS-SSIM variant). Note that HDN training in HDN$\oplus$$\mu\text{Split}$ was quite unstable and so, we had to train it with a lower hierarchy count (3 as opposed to default 6).