Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equi-ViT: Rotational Equivariant Vision Transformer for Robust Histopathology Analysis

Fuyao Chen, Yuexi Du, Elèonore V. Lieffrig, Nicha C. Dvornek, John A. Onofrey

TL;DR

Equi-ViT addresses rotation variability in histopathology by embedding rotationally equivariant features at the patch level through Gaussian Mixture Ring convolution (GMR-Conv), yielding rotation-consistent token representations in a ViT backbone. On the NCT-CRC-HE-100KK dataset, it achieves superior rotation robustness (e.g., rotation accuracy around 86.8) and demonstrates data-efficient embedding with roughly $0.79$M parameters, though CNN-based equivariant models still outperform in overall accuracy. The approach trades some raw performance for improved angular stability and efficiency, suggesting Equi-ViT as a robust backbone for future digital pathology foundation models and broader histopathology analysis. Future work includes scaling to larger datasets and integrating with radial/relative positional embeddings and rotation-aware attention biases to move toward broader equivariance with better computational efficiency.

Abstract

Vision Transformers (ViTs) have gained rapid adoption in computational pathology for their ability to model long-range dependencies through self-attention, addressing the limitations of convolutional neural networks that excel at local pattern capture but struggle with global contextual reasoning. Recent pathology-specific foundation models have further advanced performance by leveraging large-scale pretraining. However, standard ViTs remain inherently non-equivariant to transformations such as rotations and reflections, which are ubiquitous variations in histopathology imaging. To address this limitation, we propose Equi-ViT, which integrates an equivariant convolution kernel into the patch embedding stage of a ViT architecture, imparting built-in rotational equivariance to learned representations. Equi-ViT achieves superior rotation-consistent patch embeddings and stable classification performance across image orientations. Our results on a public colorectal cancer dataset demonstrate that incorporating equivariant patch embedding enhances data efficiency and robustness, suggesting that equivariant transformers could potentially serve as more generalizable backbones for the application of ViT in histopathology, such as digital pathology foundation models.

Equi-ViT: Rotational Equivariant Vision Transformer for Robust Histopathology Analysis

TL;DR

Equi-ViT addresses rotation variability in histopathology by embedding rotationally equivariant features at the patch level through Gaussian Mixture Ring convolution (GMR-Conv), yielding rotation-consistent token representations in a ViT backbone. On the NCT-CRC-HE-100KK dataset, it achieves superior rotation robustness (e.g., rotation accuracy around 86.8) and demonstrates data-efficient embedding with roughly M parameters, though CNN-based equivariant models still outperform in overall accuracy. The approach trades some raw performance for improved angular stability and efficiency, suggesting Equi-ViT as a robust backbone for future digital pathology foundation models and broader histopathology analysis. Future work includes scaling to larger datasets and integrating with radial/relative positional embeddings and rotation-aware attention biases to move toward broader equivariance with better computational efficiency.

Abstract

Vision Transformers (ViTs) have gained rapid adoption in computational pathology for their ability to model long-range dependencies through self-attention, addressing the limitations of convolutional neural networks that excel at local pattern capture but struggle with global contextual reasoning. Recent pathology-specific foundation models have further advanced performance by leveraging large-scale pretraining. However, standard ViTs remain inherently non-equivariant to transformations such as rotations and reflections, which are ubiquitous variations in histopathology imaging. To address this limitation, we propose Equi-ViT, which integrates an equivariant convolution kernel into the patch embedding stage of a ViT architecture, imparting built-in rotational equivariance to learned representations. Equi-ViT achieves superior rotation-consistent patch embeddings and stable classification performance across image orientations. Our results on a public colorectal cancer dataset demonstrate that incorporating equivariant patch embedding enhances data efficiency and robustness, suggesting that equivariant transformers could potentially serve as more generalizable backbones for the application of ViT in histopathology, such as digital pathology foundation models.
Paper Structure (8 sections, 2 figures, 2 tables)

This paper contains 8 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Experiments and Results
  4. Experimental Setup
  5. Results
  6. Discussion and Conclusion
  7. Compliance with Ethical Standards
  8. Acknowledgments

Figures (2)

  • Figure 1: Classification Accuracy Across Rotation Angles. We visualize the classification accuracy (%) per test rotation angles for both equivariant methods Equi-ViT (our approach) and ViT with R2Conv kernels Weiler2019_e2CNN (E(2) ViT) and for non-equivariant methods Standard-ViT and ViT with standard Conv2D kernels (Conv ViT). We scale the radial axis to better visualize differences in performance.
  • Figure 2: Patch Token Embedding Equivariance Analysis (a) Two example histopathology images undergoing 90°, 180°, and 270° rotation. (b) Cosine similarity was calculated between the corresponding patch token features extracted from original angle image and the rotated images, compared between standard ViT and Equi-ViT. (c) The distribution of cosine similarity values of patch tokens for standard ViT and Equi-ViT demonstrates that our equivariant feature embedding approach maintains nearly perfect patch equivariance (cosine values equal to 1) across rotations.