A 2D Semantic-Aware Position Encoding for Vision Transformers

TL;DR

This work tackles the limitation of coordinate-based position encodings in vision transformers by introducing SaPE^2, a semantic-aware 2D encoding that adapts to local content. SaPE^2 decomposes 2D positional information into horizontal and vertical 1D encodings with gates and interpolated embeddings, producing attention biases that reflect semantic relationships between patches. Empirical results on CIFAR-10/100 show SaPE^2 outperforming APE, 2D RoPE, and CoPE baselines, with notable gains when applied to the key in attention. The approach improves generalization across resolutions, enhances translation equivariance, and better aggregates features from visually similar regions, offering a promising direction for semantically aware positional encoding in vision transformers.

Abstract

Vision transformers have demonstrated significant advantages in computer vision tasks due to their ability to capture long-range dependencies and contextual relationships through self-attention. However, existing position encoding techniques, which are largely borrowed from natural language processing, fail to effectively capture semantic-aware positional relationships between image patches. Traditional approaches like absolute position encoding and relative position encoding primarily focus on 1D linear position relationship, often neglecting the semantic similarity between distant yet contextually related patches. These limitations hinder model generalization, translation equivariance, and the ability to effectively handle repetitive or structured patterns in images. In this paper, we propose 2-Dimensional Semantic-Aware Position Encoding ($\text{SaPE}^2$), a novel position encoding method with semantic awareness that dynamically adapts position representations by leveraging local content instead of fixed linear position relationship or spatial coordinates. Our method enhances the model's ability to generalize across varying image resolutions and scales, improves translation equivariance, and better aggregates features for visually similar but spatially distant patches. By integrating $\text{SaPE}^2$ into vision transformers, we bridge the gap between position encoding and perceptual similarity, thereby improving performance on computer vision tasks.

Figures (4)

• Figure 1: The overall pipeline of the vision model and a comparison between the absolute PE, relative PE, and $\text{SaPE}^2$. The patch with the red border is the target patch for relative PE and our proposed method ($\text{SaPE}^2$). The number in each patch indicates the position information of the patch. This is just a schematic diagram; please refer to Figure \ref{['fig:pipeline']} for details on $\text{SaPE}^2$.
• Figure 2: The pipeline of calculating the relative positional relationship between two patches which in the different x-axes and y-axes.
• Figure 3: Self-attention mechanism with $\text{SaPE}^2$, the position encoding can be applied to either key or query. For simplicity, we ignore the channel dimension here. The dimensions and symbols outside the parentheses indicate operations on the x-axis, while those inside indicate operations on the y-axis.
• Figure 4: Visualization of the attention bias calculated based on $\text{SaPE}^2$. The patch size is set to 4, resulting in 8×8 patches for 32×32 images. The color transition from white to green indicates further semantic distance. Left: the original image; Middle: the $\text{SaPE}^2$ value between different patches; Right: visualization of $\text{SaPE}^2$ on patch from the background and the subject.The results indicate that our position encoding method effectively helps the model perceive contextual information.