A Circular Argument : Does RoPE need to be Equivariant for Vision?

TL;DR

This work interrogates the widely held belief that RoPE’s success depends on strict shift-equivariance in vision. It shows that in 1D the RoPE mechanism with learned frequencies is equivalent to 1D-LieRE, and that in M-D settings equivariance requires commutative generators, leading to Mixed RoPE as a general solution. The authors then introduce Spherical RoPE (non-commutative) and Uniform RoPE (shared frequency) to test causality, finding that Spherical RoPE matches or outperforms equivariant alternatives, and that diversity of frequencies, rather than strict relativity, drives performance. Across CIFAR-100 and ImageNet-1K with ViT-S backbones, RoPE variants outperform Learned APE, yet Uniform RoPE can be weaker, suggesting that oblique, diverse frequencies play a crucial role. The results imply that relative positional embeddings may not be essential for strong vision performance, motivating faster, more general encodings for future work.

Abstract

Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such as images and videos. The success of RoPE has been thought to be due to its positional equivariance, i.e. its status as a relative positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for M-dimensional data, if we require commutative generators -- a property necessary for RoPE's equivariance. However, we question whether strict equivariance plays a large role in RoPE's performance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but assumes non-commutative generators. Empirically, we find Spherical RoPE to have the equivalent or better learning behavior compared to its equivariant analogues. This suggests that relative positional embeddings are not as important as is commonly believed, at least within computer vision. We expect this discovery to facilitate future work in positional encodings for vision that can be faster and generalize better by removing the preconception that they must be relative.

Figures (6)

• Figure 1: The attention patterns of Axial and Mixed RoPE. A. Each dimension pair in the query and key vectors is rotated based on the position creating an attention pattern. The pixel value of the attention pattern is $\alpha(\mathbf{x}_q,\mathbf{x}_k, \mathbf{p}, \mathbf{0})$, where $\mathbf{p} = (i,j)$ -- the pixel location. On the left, the attention pattern of individual component-pairs in the embedding vector is shown and, on the right, the components are combined into the overall attention pattern for a randomly sampled query and key vector. B. Location of the rotations frequencies in 2D frequency space. Axial RoPE can only represent frequencies that lie on an axis resulting in the grid-like attention patterns. Unlike Axial RoPE, Mixed RoPE can assign different directions to each component-pair (A Bottom). When Axial RoPE uses fixed frequencies, the frequencies are spread exponentially. However, they can be implemented as learnable parameters. For Uniform RoPE, all frequencies are fixed to a single value for each axis.
• Figure 2: Diagram of each rotary embedding's effect on the subvector, $\mathbf{z}_d$. While Mixed RoPE affects 2D vector pairs, Spherical RoPE affects 3D vector triplets. Axial RoPE rotates independent dimensions for $p_x$, thus containing pairs of pairs, or effectively quadruples. Each $\mathbf{z}$ contains $D$ sub-vectors rotating at different frequencies. While the order in which the rotations are applied does not matter for Axial or Mixed RoPE, order matters for Spherical RoPE. Explicitly, the triplet is first rotated around the axis associated with $p_x$ and then rotated around the axis associated with $p_y$.
• Figure 2: Performance comparison (top-1 accuracy) across datasets and methods.
• Figure 3: Dependence of accuracy on image resolution for ViT-S with various positional embedding methods on ImageNet-1k. Error bars reflect the standard deviation across three models trained with different seeds.
• Figure 4: The distribution of learned frequencies in each layer of the ViT. Every method tends to learn low frequency positional encodings in the later layers of the network, meaning representations in the later layers are more invariant to position.

Theorems & Definitions (5)

• theorem 1
• theorem 2
• theorem 3
• lemma 1
• lemma 2