Learning to Adapt to Position Bias in Vision Transformer Classifiers

TL;DR

This work addresses how dataset position bias affects Vision Transformer classifiers and proposes Position-SHAP (P-SHAP) to quantify the reliance on positional information by extending Kernel SHAP to position embeddings. It demonstrates that the optimal position embedding method depends on the dataset's bias level and introduces Auto-PE, a single-parameter gate that modulates the PE norm (and Auto-RoPE for RoPE) to unlearn or tune positional information. Through controlled and real-world datasets, the study shows that datasets with different bias levels benefit from different embeddings, and that Auto-PE often matches or surpasses fixed embeddings without manual tuning. The results offer a bias-aware approach to selecting and adapting position embeddings, with practical implications for improving ViT performance and enabling dataset auditing.

Abstract

How discriminative position information is for image classification depends on the data. On the one hand, the camera position is arbitrary and objects can appear anywhere in the image, arguing for translation invariance. At the same time, position information is key for exploiting capture/center bias, and scene layout, e.g.: the sky is up. We show that position bias, the level to which a dataset is more easily solved when positional information on input features is used, plays a crucial role in the performance of Vision Transformers image classifiers. To investigate, we propose Position-SHAP, a direct measure of position bias by extending SHAP to work with position embeddings. We show various levels of position bias in different datasets, and find that the optimal choice of position embedding depends on the position bias apparent in the dataset. We therefore propose Auto-PE, a single-parameter position embedding extension, which allows the position embedding to modulate its norm, enabling the unlearning of position information. Auto-PE combines with existing PEs to match or improve accuracy on classification datasets.

Figures (9)

• Figure 1: In addition to appearance information, some classification datasets require position information to be solved, and some do not. In EuroSAT, the object of interest appears anywhere in the pecture. In ImageNet, often but not always the object of interest is centered in the frame ("capture bias"). In SVHN, the task is explicitly to classify only the middle digit.
• Figure 2: CIFAR-10-Position toy dataset construction. Top row shows samples from CIFAR-10, middle row shows the possible positions and their probabilities for each class in CIFAR-10-Position, and bottom row shows samples from all classes in CIFAR-10-Position.
• Figure 5: Histogram of CIFAR-10-Position validation samples, split by correct/incorrect prediction and position in- or dependent. Samples independent of position still use position embeddings to link neighboring patches, but P-SHAP is on average higher in position-dependent samples.
• Figure 6: Reproduction of Fig. \ref{['fig:results-type3-advancedpemethods']} in Sec. \ref{['sec:pe-methods']}: advanced PE methods
• Figure 7: Results for models trained with and without pre-trained weights. Dash-dot line indicates the P-SHAP of the pre-trained checkpoint. We find that transfer learning affects position bias, slightly reducing it. Datasets with high position bias lose more accuracy by leaving out fine-tuning than datasets with low position bias.