Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

TL;DR

This work identifies a fundamental entanglement between content ('what') and position ('where') in RoPE-based transformers and introduces Polar Coordinate Position Embeddings (PoPE) to decouple these factors. PoPE represents keys/queries as complex-valued vectors with position-dependent phases and magnitude via softplus, enabling a decoupled attention score $a_{ts}^{\text{PoPE}} = \Re[\tilde{q}_t^H \tilde{k}_s]$, and optionally a learnable per-frequency bias $\delta_c$. Across diagnostic tasks and domains—music, genomics, and language—PoPE consistently outperforms RoPE in perplexity and downstream metrics, with strong zero-shot length extrapolation and robust scaling from 124M to 774M parameters. The approach yields broader frequency usage, improved data efficiency, and practical benefits for long-context sequence modeling without requiring post-hoc interpolation or fine-tuning. Overall, PoPE offers a principled, efficient improvement to positional encoding that enhances transformer performance in autoregressive settings and across diverse data domains.

Abstract

The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indicating that what and where are entangled in the popular RoPE rotary position embedding. This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embeddings or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.

Figures (8)

• Figure 1: Illustration compares how RoPE and PoPE encode relative positions via rotations of queries. Left: Three complex-valued RoPE components having magnitudes $\mu_{q_{tc}}$ (black arrows) and initial phases $\phi_{{q}_{tc}}$ (green arcs) are constructed from three pairs of embedding features (orange arrows) of the query vector $\bm{q}_t$ (gray box) at sequence position $t$. These RoPE components are then rotated by angles $t \theta_c$ (blue arcs). Right: Three magnitude components $\mu_{q_{tc}}$ (black arrows) of complex-valued PoPE components are constructed from three embedding features of the query vector $\bm{q}_t$ (gray box) by applying softplus activation. These magnitudes (complex numbers with zero phases) are then rotated by angles $t\theta_c$ (blue arcs). PoPE uses twice the number of components than RoPE as it applies rotations to each component of the query vector $\bm{q}_t$.
• Figure 2: Length extrapolation at test-time on PG-19 dataset for different model sizes. We evaluate baselines that use RoPE (red) or YaRN (yellow) against PoPE (green) which does not apply any fine-tuning or interpolation techniques and PoPE+ft (blue) which only uses fine-tuning. Sequences at test-time are multiples of 1024 up to 10240.
• Figure 3: 2-norm plotted over 2D RoPE 'chunks' of queries (left) and keys (right) in each layer of the 124M Transformer over different RoPE frequencies. Mean over 10 different Shakespeare sonnets and 12 attention heads at each layer.
• Figure 4: Magnitude of each complex-valued features of queries (left) and keys (right) in each layer of the 124M Transformer over different PoPE frequencies. Mean over 10 different Shakespeare sonnets and 12 attention heads at each layer.
• Figure 5: 2-norm plotted over 2D RoPE components of queries (left) and keys (right) in each layer of the 253M Transformer over different RoPE frequencies. Mean over 10 different Shakespeare sonnets and 16 attention heads at each layer.