Where to Attend: A Principled Vision-Centric Position Encoding with Parabolas

TL;DR

PaPE introduces a principled, parabola-based position encoding tailored for vision transformers, embedding relative position information through a sum of concave parabolas tied to token content. By combining distance decay, directionality, and context awareness, PaPE (and its rotation-invariant variant PaPE-RI) delivers broad generality across images, point clouds, videos, events, and multi-modal data, while preserving compatibility with efficient attention kernels. Empirical results show PaPE top-performing on most datasets and exceptional extrapolation to higher image resolutions, outperforming baselines by up to $10.5\%$ in absolute accuracy without interpolation. Limitations include increased inference cost and parameter overhead, motivating future work on efficiency and broader polynomial generalizations. Overall, PaPE establishes a versatile, vision-focused encoding paradigm with strong practical impact for scalable, high-resolution vision Transformers.

Abstract

We propose Parabolic Position Encoding (PaPE), a parabola-based position encoding for vision modalities in attention-based architectures. Given a set of vision tokens-such as images, point clouds, videos, or event camera streams-our objective is to encode their positions while accounting for the characteristics of vision modalities. Prior works have largely extended position encodings from 1D-sequences in language to nD-structures in vision, but only with partial account of vision characteristics. We address this gap by designing PaPE from principles distilled from prior work: translation invariance, rotation invariance (PaPE-RI), distance decay, directionality, and context awareness. We evaluate PaPE on 8 datasets that span 4 modalities. We find that either PaPE or PaPE-RI achieves the top performance on 7 out of 8 datasets. Extrapolation experiments on ImageNet-1K show that PaPE extrapolates remarkably well, improving in absolute terms by up to 10.5% over the next-best position encoding. Code is available at https://github.com/DTU-PAS/parabolic-position-encoding.

Figures (3)

• Figure 1: Parabolic Position Encoding (PaPE).(a) PaPE encodes positions using an attention bias based on a sum of learnable parabolas with the relative position between tokens as the dependent variable. (b) Our experiments show that PaPE is general---either PaPE or PaPE-RI outperforms all baselines on 7 out of 8 datasets that span 4 vision modalities. (c) PaPE has remarkable classification extrapolation, showing high robustness beyond the training resolution of $224^2$.
• Figure 2: Overview of Parabolic Position Encoding (PaPE). PaPE decomposes attention (a) into distance (b), direction (c), and semantics (d). Using the dog’s eye as the query, PaPE learns to look in a bottom-right direction, while decaying attention with distance. The attention (a) is compatible with efficient attention kernels through separate query-key transformations. Colormap:
• Figure 3: Model analysis on ImageNet-1K. Red ($z > 0$) highlights heads that lean heavily on positional information, while blue ($z < 0$) marks heads that prioritize semantic content in deciding what to attend to. Positions are used most strongly in early layers.

Theorems & Definitions (2)

• Definition 3.1
• Definition 3.2