Unpaired Image-to-Image Translation for Segmentation and Signal Unmixing

TL;DR

This work presents Ui2i, a unpaired image-to-image translation framework tailored for content-preserving translations in biomedical imaging. It replaces feature-based normalization with approximate bidirectional spectral normalization and integrates UNet-like generators with skip connections and channel-spatial attention, together with differentiable data augmentation and a cross-domain contrastive loss. The approach is demonstrated on two biomedical tasks: domain adaptation for nuclear segmentation (IHC to H&E translation enabling StarDist to segment translated images) and unmixing of single-channel immunofluorescence data to separate superimposed signals, achieving state-of-the-art-like results without paired data. The combination of architecture choices and training objectives yields improved content fidelity over CycleGAN and enables practical applications such as reduced fluorophore usage in multiplexed imaging.

Abstract

This work introduces Ui2i, a novel model for unpaired image-to-image translation, trained on content-wise unpaired datasets to enable style transfer across domains while preserving content. Building on CycleGAN, Ui2i incorporates key modifications to better disentangle content and style features, and preserve content integrity. Specifically, Ui2i employs U-Net-based generators with skip connections to propagate localized shallow features deep into the generator. Ui2i removes feature-based normalization layers from all modules and replaces them with approximate bidirectional spectral normalization -- a parameter-based alternative that enhances training stability. To further support content preservation, channel and spatial attention mechanisms are integrated into the generators. Training is facilitated through image scale augmentation. Evaluation on two biomedical tasks -- domain adaptation for nuclear segmentation in immunohistochemistry (IHC) images and unmixing of biological structures superimposed in single-channel immunofluorescence (IF) images -- demonstrates Ui2i's ability to preserve content fidelity in settings that demand more accurate structural preservation than typical translation tasks. To the best of our knowledge, Ui2i is the first approach capable of separating superimposed signals in IF images using real, unpaired training data.

Figures (14)

• Figure 1: Artifacts in image-to-image translation from the IHC to H&E domain. (a) Input IHC image. (b) CycleGAN-translated H&E image (b-left) shows numerous artifacts giving rise to false positives in the corresponding StarDist segmentation (b-right). (b) Ui2i-translated H&E image (c-left) contains significantly fewer artifacts leading to improved StarDist segmentation (c-right). In both (b) and (c), StarDist stardist is applied to the translated H&E image, and the nuclear segmentation results (marked green) overlaid on the original input image for clarity.
• Figure 2: Our Ui2i comprises two generators, $G_{AB}$ and $G_{BA}$, that use a UNet-like architecture featuring skip connections and self-attention in certain encoder blocks (marked with "Attention") to better preserve spatial content. "Residual" denotes the usage of residual connections.
• Figure 3: Convolutional block in the generator of Ui2i shown in Fig. \ref{['fig:generator']}. Depending on the block's position within the network, optional components (marked with dashed gray lines) are enabled or omitted. The first convolutional layer adapts the number of channels and receives a skip connection. The output is then passed through a number of convolutional layers (equal to 2 in our experiments) without changing the number of channels. The resulting feature map is refined by the spatial-channel attention and residual connection. ABSN stands for approximate bidirectional spectral normalization.
• Figure 4: Our Ui2i model is trained to translate IHC images to the H&E domain. Given an IHC input image (realB), the translated output (fakeA) is directly segmented by StarDist stardist, pretrained on H&E images (Seg A). Segmentation results (shown in green) are overlaid on the original input image.
• Figure 5: Breast and colon tissue unmixing results. Single-channel images, shown in the left column are separated into two channels (middle column). Example two-channel images from the training set are shown in the right column for reference.