Mask-guided cross-image attention for zero-shot in-silico histopathologic image generation with a diffusion model

TL;DR

This work addresses the high cost of annotating histopathology data by enabling zero-shot diffusion-based generation of immunohistochemistry images through mask-guided, multi-class appearance transfer. The authors extend a cross-image attention diffusion framework with class-specific AdaIN masking to generate realistic in-silico images that align with target segmentation masks, reducing manual annotation needs by about $75\%$ while maintaining segmentation performance. Quantitative results on epithelium segmentation in NSCLC show that the proposed multi-class diffusion augmentation outperforms a baseline, with additional qualitative feedback from a board-certified pathologist guiding potential improvements. The study demonstrates the practical potential of zero-shot diffusion in computational pathology for data-efficient training and model finetuning, while outlining avenues for unsupervised extension and broader stain applications.

Abstract

Creating in-silico data with generative AI promises a cost-effective alternative to staining, imaging, and annotating whole slide images in computational pathology. Diffusion models are the state-of-the-art solution for generating in-silico images, offering unparalleled fidelity and realism. Using appearance transfer diffusion models allows for zero-shot image generation, facilitating fast application and making model training unnecessary. However current appearance transfer diffusion models are designed for natural images, where the main task is to transfer the foreground object from an origin to a target domain, while the background is of insignificant importance. In computational pathology, specifically in oncology, it is however not straightforward to define which objects in an image should be classified as foreground and background, as all objects in an image may be of critical importance for the detailed understanding the tumor micro-environment. We contribute to the applicability of appearance transfer diffusion models to immunohistochemistry-stained images by modifying the appearance transfer guidance to alternate between class-specific AdaIN feature statistics matchings using existing segmentation masks. The performance of the proposed method is demonstrated on the downstream task of supervised epithelium segmentation, showing that the number of manual annotations required for model training can be reduced by 75%, outperforming the baseline approach. Additionally, we consulted with a certified pathologist to investigate future improvements. We anticipate this work to inspire the application of zero-shot diffusion models in computational pathology, providing an efficient method to generate in-silico images with unmatched fidelity and realism, which prove meaningful for downstream tasks, such as training existing deep learning models or finetuning foundation models.

Figures (4)

• Figure 1: a By swapping key, values and queries from the appearance image (${i_{source}}$) with the target structure image (${m_{target}}$) we transfer the appearance of an image to the structure of a mask, generating an image (${i_{target}}$) that matches the target mask (${m_{target}}$). This process alternates AdaIN feature statistics matching across different classes, utilizing the existing segmentation masks on the latent representations ${z^{source}}$ and ${z^{target}}$. b Real images and their corresponding in-silico images, which share the same appearance but different structures. c A UMAP plot reveals overlapping clusters of real and in-silico images.
• Figure 2: a Our model generates more realistic images than the baseline model. b From a dataset of 2,864 manually annotated images, we use a 25% subset (N=716 images) and generate four in-silico images from each image. Semantic epithelium segmentation models trained on images generated with our diffusion model (red) outperform those trained on images generated by the baseline model (petrol). The upper (blue) and lower (black) baseline segmentation models were trained on the full dataset and the 25% subset of the labeled patches, respectively. Our test-set was annotated by three independent pathologists. The mean Dice score and corresponding confidence intervals (whiskers) are displayed. c A violin plot showing Dice scores for each of the 43 FOVs and the three pathologists reveals significant differences (p $<$ 0.05, Wilcoxon signed-rank test, N=129), with statistical significance indicated by a star.
• Figure 3: An ablation study was conducted to investigate performance saturation using 10% ($N=286$ images) and 25% ($N=716$ images) subsets. The Dice scores for the segmented epithelium regions, compared against annotations from three pathologists (denoted by triangles and dotted lines), are displayed alongside the corresponding mean (solid line) and standard deviation (shaded area).
• Figure 4: A board-certified pathologist assessed the differences between real and in-silico images.