Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers

Bum Jun Kim, Sang Woo Kim

TL;DR

Vision transformers rely on absolute positional embeddings, and standard CNN-oriented augmentations can perturb input variance, causing variance-shift that undermines test-time performance. The authors provide a theoretical framework for variance propagation in ViTs, showing that maintaining consistent train/test variance ratios between patch inputs and positional embeddings is essential, and that common augmentations (Mixup, random resize crop) can break this balance while Cutmix remains safe. They propose practical remedies: rescale upsampled positional embeddings by $1/\sqrt{k}$ to align train/test variances, and configure random erasing with pixel mode and default mean-std normalization; they also caution against Mixup when training ViTs. Empirical results on ImageNet classification and semantic segmentation demonstrate that these variance-consistency guidelines yield measurable performance gains, offering actionable augmentation strategies for ViTs and clarifying when CNN-style augmentations translate well to transformer-based vision models.

Abstract

Vision transformers (ViTs) have demonstrated remarkable performance in a variety of vision tasks. Despite their promising capabilities, training a ViT requires a large amount of diverse data. Several studies empirically found that using rich data augmentations, such as Mixup, Cutmix, and random erasing, is critical to the successful training of ViTs. Now, the use of rich data augmentations has become a standard practice in the current state. However, we report a vulnerability to this practice: Certain data augmentations such as Mixup cause a variance shift in the positional embedding of ViT, which has been a hidden factor that degrades the performance of ViT during the test phase. We claim that achieving a stable effect from positional embedding requires a specific condition on the image, which is often broken for the current data augmentation methods. We provide a detailed analysis of this problem as well as the correct configuration for these data augmentations to remove the side effects of variance shift. Experiments showed that adopting our guidelines improves the performance of ViTs compared with the current configuration of data augmentations.

Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers

TL;DR

Vision transformers rely on absolute positional embeddings, and standard CNN-oriented augmentations can perturb input variance, causing variance-shift that undermines test-time performance. The authors provide a theoretical framework for variance propagation in ViTs, showing that maintaining consistent train/test variance ratios between patch inputs and positional embeddings is essential, and that common augmentations (Mixup, random resize crop) can break this balance while Cutmix remains safe. They propose practical remedies: rescale upsampled positional embeddings by to align train/test variances, and configure random erasing with pixel mode and default mean-std normalization; they also caution against Mixup when training ViTs. Empirical results on ImageNet classification and semantic segmentation demonstrate that these variance-consistency guidelines yield measurable performance gains, offering actionable augmentation strategies for ViTs and clarifying when CNN-style augmentations translate well to transformer-based vision models.

Abstract

Vision transformers (ViTs) have demonstrated remarkable performance in a variety of vision tasks. Despite their promising capabilities, training a ViT requires a large amount of diverse data. Several studies empirically found that using rich data augmentations, such as Mixup, Cutmix, and random erasing, is critical to the successful training of ViTs. Now, the use of rich data augmentations has become a standard practice in the current state. However, we report a vulnerability to this practice: Certain data augmentations such as Mixup cause a variance shift in the positional embedding of ViT, which has been a hidden factor that degrades the performance of ViT during the test phase. We claim that achieving a stable effect from positional embedding requires a specific condition on the image, which is often broken for the current data augmentation methods. We provide a detailed analysis of this problem as well as the correct configuration for these data augmentations to remove the side effects of variance shift. Experiments showed that adopting our guidelines improves the performance of ViTs compared with the current configuration of data augmentations.
Paper Structure (28 sections, 6 theorems, 5 equations, 4 figures, 8 tables)

This paper contains 28 sections, 6 theorems, 5 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Theory and Practice: Data Augmentations for Vision Transformers
  3. Problem Statement: Variance Shift in Positional Embedding
  4. Properties of Variance
  5. Variance Shift Due to Upsampling
  6. Image Classification Scenario
  7. Semantic Segmentation Scenario
  8. Variance Shift Due to Mixup
  9. Variance Shift Due to Random Erasing
  10. Experiments
  11. Rescaling Positional Embedding
  12. ImageNet Evaluation
  13. Semantic Segmentation
  14. Effect of Mixup and Cutmix
  15. ImageNet Training
  16. ...and 13 more sections

Key Result

Theorem 1

Upsampling yields $\mathop{\mathrm{E}}\nolimits[\mathop{\mathrm{UP}}\nolimits(\mathbf{v})] = \mathop{\mathrm{E}}\nolimits[\mathbf{v}]$ and $\mathop{\mathrm{Var}}\nolimits[\mathop{\mathrm{UP}}\nolimits(\mathbf{v})] = \mathop{\mathrm{Var}}\nolimits[\mathbf{v}]$ for duplication-type upsampling but not

Figures (4)

  • Figure 1: Overview of the early stage of ViT. Variance is depicted by chroma. If the input image or patch embedding exhibits different variances during the training (left) and test (right) phases, positional embedding inconsistently affects the output.
  • Figure 2: Positional embedding is upsampled depending on the size of patch embedding, which causes inconsistent variance of positional embedding
  • Figure 3: Illustration of Mixup, Cutmix, and random erasing. The level of darkness indicates variance.
  • Figure 4: Illustration of the coordinate and interpolation

Theorems & Definitions (6)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Lemma 1
  • Proposition 4