Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

Adam Pardyl, Grzegorz Kurzejamski, Jan Olszewski, Tomasz Trzciński, Bartosz Zieliński

TL;DR

This paper tackles the rigidity of Vision Transformers' input sampling by introducing the notion of input elasticity and a protocol to evaluate resilience to scale, position, and missing data perturbations. It then proposes ElasticViT, featuring a 4D sine-cosine encoding for patch coordinates and a training regime with PatchMix augmentations to enhance elasticity. Through extensive experiments across ImageNet-1k and transfer tasks, ElasticViT demonstrates superior robustness to elastic inputs and maintains competitive accuracy under perturbations, outperforming vanilla ViT and other baselines in many perturbed scenarios. The work also explores adaptive patch sampling (CENTRAL and EDGE) and reveals promising directions for real-world embodied AI applications where observations are partial, multi-scale, and non-grid aligned.

Abstract

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.

Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers

TL;DR

This paper tackles the rigidity of Vision Transformers' input sampling by introducing the notion of input elasticity and a protocol to evaluate resilience to scale, position, and missing data perturbations. It then proposes ElasticViT, featuring a 4D sine-cosine encoding for patch coordinates and a training regime with PatchMix augmentations to enhance elasticity. Through extensive experiments across ImageNet-1k and transfer tasks, ElasticViT demonstrates superior robustness to elastic inputs and maintains competitive accuracy under perturbations, outperforming vanilla ViT and other baselines in many perturbed scenarios. The work also explores adaptive patch sampling (CENTRAL and EDGE) and reveals promising directions for real-world embodied AI applications where observations are partial, multi-scale, and non-grid aligned.

Abstract

Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.
Paper Structure (18 sections, 2 equations, 9 figures)

This paper contains 18 sections, 2 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation protocol
  4. Elasticities
  5. Protocol
  6. Elastic ViT
  7. Position and scale encoding
  8. Augmented training
  9. Experimental setup
  10. Research questions
  11. Scale elasticity
  12. Missing data elasticity
  13. Positional elasticity
  14. Combining perturbations
  15. Patch sampling strategies
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Grid vs. elastic sampling: patch sampling with arbitrary patch positions and scales creates new possibilities for more effective and efficient vision transformers.
  • Figure 2: Evaluation protocol: To analyze scale, position, and missing data elasticity we introduce three types of perturbations that are applied separately to each patch from the sampling grid.
  • Figure 3: PatchMix: We introduce PatchMix, an adaptation of CutMix and TokenMix to elasticity oriented training regime. PatchMix takes full advantage of the ElasticViT position and scale encoding, mixing randomly sampled patches.
  • Figure 4: Isolated perturbations:(a) The impact of changing the patch scale on accuracy. ElasticViT can extract information from patches of various scales, being superior to other models in situations when information is lost either because patches do not cover the whole image or because the patches are of low resolution. (b) The effect of random patch dropout on the accuracy. ElasticViT is more resilient to significant patch dropout than the other models and even outperforms the image reconstruction model (MAE) at the end of the measured spectrum. (c) The effect of applying positional perturbation on the accuracy. ElasticViT can naturally interpret a whole range of possible position values. Being almost immune to the movement of sampled patches it eventually outperforms all but MAE for large perturbations. Note that the X-axis represents the percentage of patch movement relative to the patch size
  • Figure 5: Combining perturbations: The results of mixing multiple types of perturbations in one experiment. We observe that the performance of baseline models degrades quickly as perturbations add up. At the same time, the accuracy of ElasticViT remains stable for much longer, allowing more elastic input.
  • ...and 4 more figures