Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAINT: Pathology-Aware Integrated Next-Scale Transformation for Virtual Immunohistochemistry

Rongze Ma, Mengkang Lu, Zhenyu Xiang, Yongsheng Pan, Yicheng Wu, Qingjie Zeng, Yong Xia

TL;DR

PAINT reframes virtual IHC as a structure-aware autoregressive problem, addressing the ill-posed mapping from H&E morphology to protein expression. By learning modality-specific VQ-VAE representations, aligning cross-modal latent and image spaces with LSA and ISA, and initializing autoregressive synthesis with a morphology-grounded 3S-Map, PAINT achieves superior structural fidelity and clinically relevant performance on IHC4BC and MIST. The three-stage pipeline—dual VQ-VAE representations, cross-modal translation, and next-scale autoregressive generation—yields state-of-the-art results in both image quality and downstream diagnostic tasks, while ablation confirms the critical role of the structural priors and multi-scale guidance. This structure-aware framework offers a principled path forward for reliable, clinically interpretable virtual staining in computational pathology.

Abstract

Virtual immunohistochemistry (IHC) aims to computationally synthesize molecular staining patterns from routine Hematoxylin and Eosin (H\&E) images, offering a cost-effective and tissue-efficient alternative to traditional physical staining. However, this task is particularly challenging: H\&E morphology provides ambiguous cues about protein expression, and similar tissue structures may correspond to distinct molecular states. Most existing methods focus on direct appearance synthesis to implicitly achieve cross-modal generation, often resulting in semantic inconsistencies due to insufficient structural priors. In this paper, we propose Pathology-Aware Integrated Next-Scale Transformation (PAINT), a visual autoregressive framework that reformulates the synthesis process as a structure-first conditional generation task. Unlike direct image translation, PAINT enforces a causal order by resolving molecular details conditioned on a global structural layout. Central to this approach is the introduction of a Spatial Structural Start Map (3S-Map), which grounds the autoregressive initialization in observed morphology, ensuring deterministic, spatially aligned synthesis. Experiments on the IHC4BC and MIST datasets demonstrate that PAINT outperforms state-of-the-art methods in structural fidelity and clinical downstream tasks, validating the potential of structure-guided autoregressive modeling.

PAINT: Pathology-Aware Integrated Next-Scale Transformation for Virtual Immunohistochemistry

TL;DR

PAINT reframes virtual IHC as a structure-aware autoregressive problem, addressing the ill-posed mapping from H&E morphology to protein expression. By learning modality-specific VQ-VAE representations, aligning cross-modal latent and image spaces with LSA and ISA, and initializing autoregressive synthesis with a morphology-grounded 3S-Map, PAINT achieves superior structural fidelity and clinically relevant performance on IHC4BC and MIST. The three-stage pipeline—dual VQ-VAE representations, cross-modal translation, and next-scale autoregressive generation—yields state-of-the-art results in both image quality and downstream diagnostic tasks, while ablation confirms the critical role of the structural priors and multi-scale guidance. This structure-aware framework offers a principled path forward for reliable, clinically interpretable virtual staining in computational pathology.

Abstract

Virtual immunohistochemistry (IHC) aims to computationally synthesize molecular staining patterns from routine Hematoxylin and Eosin (H\&E) images, offering a cost-effective and tissue-efficient alternative to traditional physical staining. However, this task is particularly challenging: H\&E morphology provides ambiguous cues about protein expression, and similar tissue structures may correspond to distinct molecular states. Most existing methods focus on direct appearance synthesis to implicitly achieve cross-modal generation, often resulting in semantic inconsistencies due to insufficient structural priors. In this paper, we propose Pathology-Aware Integrated Next-Scale Transformation (PAINT), a visual autoregressive framework that reformulates the synthesis process as a structure-first conditional generation task. Unlike direct image translation, PAINT enforces a causal order by resolving molecular details conditioned on a global structural layout. Central to this approach is the introduction of a Spatial Structural Start Map (3S-Map), which grounds the autoregressive initialization in observed morphology, ensuring deterministic, spatially aligned synthesis. Experiments on the IHC4BC and MIST datasets demonstrate that PAINT outperforms state-of-the-art methods in structural fidelity and clinical downstream tasks, validating the potential of structure-guided autoregressive modeling.
Paper Structure (21 sections, 12 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 21 sections, 12 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Models for Image Synthesis and Translation
  4. Virtual Staining in Computational Pathology
  5. Methodology
  6. Overview
  7. Initialization of modality-specific visual perception
  8. Feature Alignment for 3S-Map Initialization
  9. Visual Autoregressive Generation via Structural Conditioning
  10. Training Objective and Inference
  11. Experiments
  12. Datasets
  13. Evaluation Metrics
  14. Implementation Details
  15. Experimental Results
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Standard VAR (top) relies on a generic token (Start of Sequence [SOS]) lacking morphological grounding. In contrast, PAINT (bottom) introduces a Spatial Structural Start Map ($S_{start}$)—a dense feature prior synthesized from the H&E image—that strictly aligns the autoregressive synthesis with the source morphology.
  • Figure 2: Overview of the proposed PAINT framework. (a) PAINT independently trains two domain-specific multi-scale VQ-VAEs for H&E and IHC images. Each encoder maps the input image to continuous latent features, which are discretized into hierarchical token maps via residual vector quantization. The corresponding decoders reconstruct the images to ensure compact yet high-fidelity representations. After training, all VQ-VAE encoders and decoders are frozen. (b) Using paired H&E--IHC samples registered by DeeperHistReg, a translation network (U-Net) maps H&E latent features, extracted by the frozen H&E encoder, into the continuous IHC feature space. This stage is supervised by LSA for feature consistency and ISA via the frozen IHC decoder for reconstruction fidelity, producing spatially aligned and structurally consistent intermediate representations. (c) A Visual Autoregressive (VAR) Transformer progressively predicts IHC token maps in a coarse-to-fine, next-scale manner. Translated features are injected as structural conditions via Adaptive Layer Normalization (AdaLN), guiding the autoregressive process to preserve tissue morphology while synthesizing realistic molecular textures. The final IHC image is reconstructed by the IHC VQ-VAE decoder.
  • Figure 3: Visual comparison of ER, HER2, Ki67 and PR generation. From top to bottom, the rows represent different IHC markers. The leftmost column shows the input H&E, and the rightmost column shows the Real IHC.