Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Selective Rotary Position Embedding

Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher

TL;DR

Selective RoPE introduces an input-dependent rotary position embedding that generalizes RoPE to learnable angles and integrates with gating to provide both rotation and decay in sequence models. By viewing softmax attention through an RFF lens, the authors show implicit rotations exist but require decay to avoid spectral leakage, and they argue that combining both components yields greater recall and expressivity. The proposed Selective RoPE defines a RoPE-compatible state update that applies rotations to queries/keys with learnable frequencies while leveraging gates for forgetting, enabling efficient real-valued implementation. Empirically, integrating Selective RoPE into GLA, Gated DeltaNet, and FoX improves recall-focused synthetic tasks (MQAR, MAD, state tracking) and downstream language modeling with modest computational overhead. This work unifies rotation and decay as core principles for memory and relative-position encoding in Transformers and suggests avenues for future work on length extrapolation, gate design, and interaction with RoPE variants.

Abstract

Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \textit{Selective RoPE}, an \textit{input-dependent} rotary embedding mechanism, that generalizes \textit{RoPE}, and enables rotation in \textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.

Selective Rotary Position Embedding

TL;DR

Selective RoPE introduces an input-dependent rotary position embedding that generalizes RoPE to learnable angles and integrates with gating to provide both rotation and decay in sequence models. By viewing softmax attention through an RFF lens, the authors show implicit rotations exist but require decay to avoid spectral leakage, and they argue that combining both components yields greater recall and expressivity. The proposed Selective RoPE defines a RoPE-compatible state update that applies rotations to queries/keys with learnable frequencies while leveraging gates for forgetting, enabling efficient real-valued implementation. Empirically, integrating Selective RoPE into GLA, Gated DeltaNet, and FoX improves recall-focused synthetic tasks (MQAR, MAD, state tracking) and downstream language modeling with modest computational overhead. This work unifies rotation and decay as core principles for memory and relative-position encoding in Transformers and suggests avenues for future work on length extrapolation, gate design, and interaction with RoPE variants.

Abstract

Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \textit{Selective RoPE}, an \textit{input-dependent} rotary embedding mechanism, that generalizes \textit{RoPE}, and enables rotation in \textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.

Paper Structure

This paper contains 39 sections, 1 theorem, 50 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Our view: recall needs rotation and decay.
  3. Why rotation alone is insufficient.
  4. Background
  5. Transformers.
  6. RoPE and Complex Linear Attention.
  7. A unifying view: Decay and Rotation
  8. Softmax attention implicitly performs rotations
  9. Necessity of Gating: Spectral Leakage in Diagonal SSMs
  10. Design Principles for Linear Attention
  11. Experiments
  12. Implementation
  13. Synthetic Language Tasks
  14. MQAR.
  15. MAD and Copying.
  16. ...and 24 more sections

Key Result

Theorem 1

Let the expected error of the RFF kernel over ${\bm{\omega}}_j\sim \mathcal{N}\left(0, \sigma^2{\bm{I}}\right)$ be as follows: $\text{ERR}\left[{\bm{q}}_t, {\bm{k}}_\tau\right]= \mathbb{E}_{{\bm{\omega}}_j}\left[\left(\frac{1}{D}\sum_{j=1}^D \phi_{{\bm{\omega}}_j}({\bm{q}}_t)\cdot \phi_{{\bm{\omega}

Figures (10)

  • Figure 1: Our methods (right two columns) are highlighted with a light blue background. Left to right: GLA, RoPE, Selective RoPE (ours), Selective RoPE + Decay (ours). As we observe, the forget gate only encodes positional information through scale. On the other hand, both RoPE and Selective RoPE allow for positional information to be encoded through rotation, with the selective variant taking advantage of arbitrary angles. Combining the two methods yields the best results.
  • Figure 2: The distribution of the phase temperatures in RoPE vs. Selective RoPE. $\epsilon$ is the inverse of the RoPE base frequency and the upper-bound of query-key angle in our temperature. Details about the parameterization available in \ref{['app:temp-param']}.
  • Figure 3: The effects of windowing on the spectrogram of a finite sample of a sequence.
  • Figure 4: Pseudocode of Selective RoPE.
  • Figure 5: Prefill throughput on NVIDIA B200 with batch size=1
  • ...and 5 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Proof 1