Self2Seg: Single-Image Self-Supervised Joint Segmentation and Denoising

TL;DR

Self2Seg proposes a self-supervised, single-image framework that jointly denoises and segments an image by coupling a variational Chan–Vese–style energy with region-specific denoisers. It alternatingly trains foreground and background denoisers (via a Noise2Fast-inspired strategy) and updates the segmentation mask through a convexified, primal-dual scheme, with theoretical guarantees on monotone energy reduction and convergence to stationary points. The approach yields improved segmentation accuracy and denoising quality on noisy microscopy data compared to sequential or purely learning-based baselines, and it naturally extends to multi-class settings. By enabling region-adaptive restoration without labeled data, it offers a practical, data-efficient tool for biomedical imaging and related domains.

Abstract

We develop Self2Seg, a self-supervised method for the joint segmentation and denoising of a single image. To this end, we combine the advantages of variational segmentation with self-supervised deep learning. One major benefit of our method lies in the fact, that in contrast to data-driven methods, where huge amounts of labeled samples are necessary, Self2Seg segments an image into meaningful regions without any training database. Moreover, we demonstrate that self-supervised denoising itself is significantly improved through the region-specific learning of Self2Seg. Therefore, we introduce a novel self-supervised energy functional in which denoising and segmentation are coupled in a way that both tasks benefit from each other. We propose a unified optimisation strategy and numerically show that for noisy microscopy images our proposed joint approach outperforms its sequential counterpart as well as alternative methods focused purely on denoising or segmentation.

Figures (10)

• Figure 1: Visualisation of the idea behind the proposed joint denoising and segmentation model. Here, we trained two networks consisting of one single filter using the Noise2Fast lehtinen2018noise2noise strategy and restricted the training to the two boxes marked in the noisy image $f$. From the two right binary images in the bottom row, we observe that the two denoising experts perform much better in the region they have been trained on. The difference images (noisy image minus Denoised by Expert 1 (resp. 2) can then be used in the segmentation process, by exploiting the fact that regions with a small denoising error for the first (resp. second) expert can be assigned as foreground (resp. background).
• Figure 2: Given noisy RGB input image (corrupted with Gaussian noise, noise level = 0.75), denoised image using Noise2Fast on the whole image, region-specific experts, and ground truth image. We clearly observe sharper edges, and better recovered color information in the "two-experts"-example.
• Figure 3: The first image shows the given greyscale input image $f$, and user defined boxes representing rough foreground and background regions. The third image highlights pixels where the foreground expert denoiser performs better than the background one, while the last image is the segmentation result obtained by minimising the proposed energy \ref{['eq:joint2']}.
• Figure 4: Alternating optimisation scheme. As a first step, regions are provided for the training of the two denoising experts using the strategy. These regions can be obtained by thresholding image values or by manually choosing boxes. The differences between the given noisy image $f$ and network outputs $\mathbf{D}_F(f)$ and $\mathbf{D}_B(f)$, are used in the subsequent segmentation step, minimising $\mathcal{E}_{\lambda,f}(\cdot, \boldsymbol{W})$ with Algorithm \ref{['alg2']}.
• Figure 5: Visual comparison of the segmentation results of data with noise level 10. From left to right, this figure shows: the noisy input, the results obtained with the proposed joint approach, the sequential approach, the chan-Vese baseline and the ground truth segmentation masks. For all compared methods, the $\lambda$ maximising the Dice score has been selected.

Theorems & Definitions (7)

• Example 1
• Remark 1: Monotonicity of alternating minimisation
• Theorem 3: Convergence of Algorithm \ref{['algo_alternate']}
• proof
• Remark 2
• Theorem 4: Thresholding
• proof