Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alias-Free ViT: Fractional Shift Invariance via Linear Attention

Hagay Michaeli, Daniel Soudry

TL;DR

This work tackles the lack of translation invariance in Vision Transformers by introducing Alias-Free Vision Transformer (AFT), which combines alias-free downsampling and nonlinearities with shift-equivariant linear attention, specifically cross-covariance attention. The key idea is to maintain shift-equivariance across patch embedding, attention, and MLP components, enabling a near-invariant global representation while preserving competitive accuracy. Empirically, AFT achieves ImageNet-level performance comparable to baselines but with substantially higher shift-consistency (≈99% for integer and half-pixel shifts) and stronger robustness to adversarial translations and realistic shifts, albeit with higher runtime due to FFT-based operations. The approach advances robust, translation-insensitive ViTs and offers a principled path to combining anti-aliasing concepts with transformer architectures in vision tasks.

Abstract

Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance. Specifically, Vision Transformers (ViTs) are not translation-invariant and are more sensitive to minor image translations than standard convnets. Previous studies have shown, however, that convnets are also not perfectly shift-invariant, due to aliasing in downsampling and nonlinear layers. Consequently, anti-aliasing approaches have been proposed to certify convnets' translation robustness. Building on this line of work, we propose an Alias-Free ViT, which combines two main components. First, it uses alias-free downsampling and nonlinearities. Second, it uses linear cross-covariance attention that is shift-equivariant to both integer and fractional translations, enabling a shift-invariant global representation. Our model maintains competitive performance in image classification and outperforms similar-sized models in terms of robustness to adversarial translations.

Alias-Free ViT: Fractional Shift Invariance via Linear Attention

TL;DR

This work tackles the lack of translation invariance in Vision Transformers by introducing Alias-Free Vision Transformer (AFT), which combines alias-free downsampling and nonlinearities with shift-equivariant linear attention, specifically cross-covariance attention. The key idea is to maintain shift-equivariance across patch embedding, attention, and MLP components, enabling a near-invariant global representation while preserving competitive accuracy. Empirically, AFT achieves ImageNet-level performance comparable to baselines but with substantially higher shift-consistency (≈99% for integer and half-pixel shifts) and stronger robustness to adversarial translations and realistic shifts, albeit with higher runtime due to FFT-based operations. The approach advances robust, translation-insensitive ViTs and offers a principled path to combining anti-aliasing concepts with transformer architectures in vision tasks.

Abstract

Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance. Specifically, Vision Transformers (ViTs) are not translation-invariant and are more sensitive to minor image translations than standard convnets. Previous studies have shown, however, that convnets are also not perfectly shift-invariant, due to aliasing in downsampling and nonlinear layers. Consequently, anti-aliasing approaches have been proposed to certify convnets' translation robustness. Building on this line of work, we propose an Alias-Free ViT, which combines two main components. First, it uses alias-free downsampling and nonlinearities. Second, it uses linear cross-covariance attention that is shift-equivariant to both integer and fractional translations, enabling a shift-invariant global representation. Our model maintains competitive performance in image classification and outperforms similar-sized models in terms of robustness to adversarial translations.
Paper Structure (42 sections, 4 theorems, 19 equations, 5 figures, 8 tables)

This paper contains 42 sections, 4 theorems, 19 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Methods
  4. Preliminaries: the Vision Transformer
  5. Multi‑Head Self‑Attention.
  6. Feed‑Forward Network.
  7. Linear Attention
  8. Alias-Free Vision Transformer
  9. Shift invariance and equivariance.
  10. Patch embedding.
  11. Positional encoding.
  12. Shift-Equivariant Attention.
  13. MLP.
  14. Layer normalization.
  15. Class token.
  16. ...and 27 more sections

Key Result

Proposition 1

$Q$, $K$ and $V$ are shift‑equivariant.

Figures (5)

  • Figure 1: Overview of the Alias-Free Vision Transformer (AFT) architecture. The input image is first processed by an alias-free patch embedding module composed of convolutional layers (Conv), batch normalization (BN), and alias-free activation, composed of upsampling (AF-Up), GELU and downsampling (AF-Down). The result is reshaped to a token matrix form and fed through $L$ Alias-Free Transformer blocks, each consisting of alias-free layer normalization (AF-LN), cross-covariance attention (XCA), alias-free local patch interaction (AF-LPI), and alias-free MLP (AF-MLP) layers, interconnected by residual connections. The result is concatenated with a learnable class token embedding and fed into two Alias-Free Class Attention blocks composed of an XCA layer and an MLP applied on the class token. The final representation is the updated class token, which is fed into a final linear classifier. Detailed explanations of each component are provided in \ref{['sec:implementation']}.
  • Figure 2: ImageNet adversarial accuracy under realistic translations. Adversarial accuracies under (a) "Crop-shifts," simulating camera translations, and (b) "Bilinear fractional shifts," simulating realistic sub-pixel image translations. The Alias-Free Transformer (AFT) consistently outperforms baseline XCiT, APS, and other vision transformer variants (CvT, Swin and ViT), demonstrating superior robustness against realistic translations.
  • Figure 2: Ablation study on alias‐free components of XCiT‐Nano (ImageNet). Evaluation of isolated alias‐free modifications to the baseline model. Alias‐free layer normalization (AF‐LayerNorm) and replacing class‐attention with average pooling (AvgPool) result in notable accuracy degradation individually. Removing positional encoding slightly improves performance. The final combined alias‐free model retains near‐baseline accuracy.
  • Figure 3: Visualization of cyclic shifts. (a) Original ImageNet deng_imagenet_2009 validation-set image. (b) Circular shift of 16 pixels in horizontal and vertical axes. (c) Circular shift of 16.5 pixels in horizontal and vertical axes. The original image is upsampled by a factor 2, circularly shifted by 33 pixels, and downsampled by factor 2.
  • Figure 4: Visualization of realistic shifts. (a) Original ImageNet deng_imagenet_2009 validation-set image --- $224 \times 224$ center crop of the original $256 \times 256$ image. (b) Crop-shift of the original image of 16 pixels in the horizontal and vertical axes. The cropped area is shifted by 16 pixels with respect to the cropped area in the original image. (c) Bilinear fractional shift of 0.5 pixels in horizontal and vertical axes. We use a $226 \times 226$ center crop of the original $256 \times 256$ image and simulate a fractional-pixel shift using a grid-sample with a fractional offset.

Theorems & Definitions (8)

  • Proposition 1
  • proof
  • Proposition 2
  • Proposition 3
  • proof
  • proof
  • Proposition 4
  • proof