Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization

TL;DR

The paper identifies spectral rigidity in fixed Rotary Positional Embeddings (RoPE) as a core bottleneck for deep algorithmic reasoning, framing a Structure Gap between content and structure. It proposes Bifocal Attention, combining Geometric Eyes (standard RoPE) with Spectral Eyes (a learnable spectral basis) and a Spectral Evolution training protocol to adapt frequencies, amplitudes, and phases during learning. The Spectral-RoPE engine is integrated surgically into transformer attention, enabling orthogonality in phase space and a harmonic representation of recursion depths. Empirical results on formal-language tasks show near-zero losses where baselines fail, with strong convergence signals and evidence of non-monotonic phase dynamics, suggesting a path toward logic-native architectures with practical gains in algorithmic reasoning and long-context generalization.

Abstract

Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation. However, we identify a significant limitation we term ''Spectral Rigidity'': standard RoPE utilizes a fixed geometric decay ($θ^{-i}$) optimized for local syntactic coherence, which fails to capture the long-range, periodic structures inherent in recursive logic and algorithmic reasoning. This results in a ''Structure Gap'', where models trained on shallow reasoning chains fail to extrapolate to deeper recursive steps. In this work, we introduce Bifocal Attention, an architectural paradigm that decouples positional encoding into two distinct modalities: Geometric Eyes (Standard RoPE) for precise token-level manipulation, and Spectral Eyes (Learnable Harmonic Operators) for tracking long-range recursive depth. We propose a novel training protocol, Spectral Evolution, which initializes positional frequencies as static geometric parameters but allows them to evolve via gradient descent into a harmonic basis optimized for the specific algorithmic topology of the task.

Figures (5)

• Figure 1: The Wavelength Mismatch Problem. (A) Standard RoPE utilizes a fixed geometric basis. At long distances (e.g., $N=60$), the fixed frequency often aligns poorly with the target token, creating a "Structure Gap". (B) Spectral-RoPE allows the frequency $\theta$ to evolve. The model discovers a resonant frequency such that $\cos(\theta \cdot N) \approx 1$, creating a "Harmonic Bridge" (Blue Dots) that connects the start and end of the recursive structure.
• Figure 2: Architecture of the Spectral-RoPE Engine. (a) Standard RoPE applies a fixed rotation. (b) Spectral-RoPE adjusts frequency (tempo). (c) Amplitude modulation amplifies signal over noise. (d) Phase shifting allows precise alignment.
• Figure 3: Topological Geometry of Attention Heads. (A) Standard RoPE representations suffer from Manifold Collapse; recursion depths cluster into generic blobs. (B) Spectral-RoPE unfurls the representation into a Harmonic Spiral. The attention mechanism maps recursion depth to a continuous 1D manifold, enabling precise differentiation of depth.
• Figure 4: Loss Landscape Comparison (Bio-Rotation Task). Standard RoPE (Grey) hits an "Entropy Floor" early ($Loss \approx 1.07$). Spectral RoPE (Blue) exhibits a "Lock-In" event at Step 150, where the evolved frequencies resonate with the sequence length, driving loss to zero.
• Figure 5: The Resonant Frequency Effect. (A) Standard RoPE acts like a radio on a static channel, often clashing with data periodicity. (B) Spectral-RoPE acts as a tunable receiver. $\Omega$ evolves to resonate with the data's inherent periodicity, allowing strong attention across vast distances.