Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

TL;DR

The paper investigates how internal attention dynamics reveal structured reasoning in decoder-only LLMs. By separating local and global attention patterns, it introduces WAAD and FAI as metrics to capture preplanning and anchoring behaviors, uncovering a recurring preplan-and-anchor rhythm. It then leverages these signals to design three structure-aware RL strategies that reweight token-level credit toward critical reasoning nodes, achieving consistent improvements across diverse reasoning tasks. The work demonstrates that aligning optimization with the model’s intrinsic reasoning rhythm enhances data efficiency, interpretability, and performance, while remaining compatible with existing RLVR workflows.

Abstract

The reasoning pattern of Large language models (LLMs) remains opaque, and Reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work positions attention as a privileged substrate that renders the internal logic of LLMs legible, not merely as a byproduct of computation, but as a mechanistic blueprint of reasoning itself. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We formalize these with two metrics: 1) Windowed Average Attention Distance, which measures the extent of backward attention within a clipped window; 2) Future Attention Influence, which quantifies a token's global importance as the average attention it receives from subsequent tokens. Taken together, these signals reveal a recurring preplan-and-anchor mechanism, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks. By aligning optimization with the model's intrinsic reasoning rhythm, we aim to transform opaque optimization into an actionable structure-aware process, hoping to offer a potential step toward more transparent and effective optimization of LLM reasoning.

Figures (10)

• Figure 1: Attention dynamics uncover reasoning patterns: Local heads exhibit a near-diagonal, sawtooth pattern that forms phrasal chunks, while global heads highlight tokens with strong downstream influence. Their coupling reveals a preplan-and-anchor rhythm: at chunk onsets, preplan tokens set up the next reasoning step, followed (or coinciding) by anchor tokens that are persistently revisited by future tokens to guide downstream reasoning. Reasoning rhythm enhances RL: Leveraging detected preplan and anchor tokens, our RL method amplifies their token-level credit accordingly, focusing learning on the critical nodes that drive more effective reasoning.
• Figure 2: Visualization and quantification of local and global attention patterns on a math reasoning case. Left: a local-focused attention map highlights near-diagonal sawtooth patterns, and a global-focused map highlights vertical stripes indicative of broader attention. Right: the corresponding token-level metrics, including FAI (Def. \ref{['def:fai']}) for receiver and global heads, WAAD (Def. \ref{['def:waad']}), and token entropy, with key points correlating with each other.
• Figure 3: Jaccard similarity of rollouts for top- and bottom-$k$ ranked by FAI. $\Pr(\text{Top} < \text{Bottom})$ denotes the percentage of trials where top-$k$ similarity is lower.
• Figure 4: An example comparing the effects of perturbing high-FAI versus low-FAI positions and continuing the rollout of the subsequent reasoning trajectory. Perturbing a high-FAI position typically leads to a clear shift in the overall reasoning logic, whereas replacing a low-FAI token more frequently just alters local phrasing, leaving the downstream reasoning trajectory essentially unchanged.
• Figure 5: Results on the Countdown and QA datasets. Bold denotes the best results per dataset.

Theorems & Definitions (2)

• definition 1
• definition 2