CellStyle: Improved Zero-Shot Cell Segmentation via Style Transfer

TL;DR

CellStyle tackles the domain-generalization problem in cell instance segmentation by enabling zero-shot adaptation to unseen microscopy datasets. It uses a diffusion-based style transfer backbone to map target dataset attributes onto annotated source images while preserving cell shapes, producing styled data $(X_{sty}, M^{GT}_{src})$ for finetuning. A cell size ratio $r$ aligns scales between source and target, and an adaptive attention scaling ratio $\alpha$ per dataset pair balances style transfer, enhancing cross-domain compatibility. Across multiple dataset pairs and segmentation models, CellStyle yields significant zero-shot improvements over baselines and competitive results with generative-model methods, with code to be released publicly.

Abstract

Cell microscopy data are abundant; however, corresponding segmentation annotations remain scarce. Moreover, variations in cell types, imaging devices, and staining techniques introduce significant domain gaps between datasets. As a result, even large, pretrained segmentation models trained on diverse datasets (source datasets) struggle to generalize to unseen datasets (target datasets). To overcome this generalization problem, we propose CellStyle, which improves the segmentation quality of such models without requiring labels for the target dataset, thereby enabling zero-shot adaptation. CellStyle transfers the attributes of an unannotated target dataset, such as texture, color, and noise, to the annotated source dataset. This transfer is performed while preserving the cell shapes of the source images, ensuring that the existing source annotations can still be used while maintaining the visual characteristics of the target dataset. The styled synthetic images with the existing annotations enable the finetuning of a generalist segmentation model for application to the unannotated target data. We demonstrate that CellStyle significantly improves zero-shot cell segmentation performance across diverse datasets by finetuning multiple segmentation models on the style-transferred data. The code will be made publicly available.

Figures (2)

• Figure 1: Overview of the CellStyle pipeline: (a) Cell Size Matching: the average cell length in $X_{\mathrm{tgt}}$ is estimated using a pretrained segmentation model and compared to $X_{\mathrm{src}}$, to compute the cell size ratio $r$ which is used to scale $x_{\mathrm{tgt}}$; (b) Style Transfer: the pretrained diffusion model generates $x_{\mathrm{sty}}$ based on $x_{\mathrm{tgt}}$ and $x_{\mathrm{src}}$; (c) Downstream task: $X_{\mathrm{sty}}$ and ground truth labels $M^{\mathrm{GT}}_{\mathrm{src}}$ from $X_{\mathrm{tgt}}$ are used to finetune segmentation models.
• Figure 2: Sample qualitative results ($x_{\mathrm{sty}}$) for each pair along with the corresponding source ($x_{\mathrm{src}}$) and the target ($x_{\mathrm{tgt}}$) images.