Attention Projection Mixing and Exogenous Anchors

TL;DR

ExoFormer, which learns dedicated exogenous anchor projections outside the sequential layer stack, decoupling the anchor role from computational refinement, is proposed, which learns dedicated exogenous anchor projections outside the sequential layer stack, allowing layers to specialize exclusively in computational refinement.

Abstract

Transformers that reuse early-layer attention projections as residuals face a fundamental tension: the first layer must simultaneously serve as a stable reference for all deeper layers and as an effective computational block. To resolve this, we propose ExoFormer, which learns dedicated exogenous anchor projections outside the sequential layer stack, decoupling the anchor role from computational refinement. Through a unified normalized mixing framework (studying different coefficient granularities: elementwise, headwise, scalar) across all attention pathways (queries, keys, values, and gate logits), ExoFormer variants consistently outperform their internal-anchor counterparts. Moreover, the dynamic variant achieves a 2.13-point increase in downstream accuracy over the baseline and demonstrates superior data efficiency, matching baseline validation loss with 1.84x fewer tokens. ExoFormer also achieves a 2x reduction in attention sink compared to standard Gated Attention. Paradoxically, all ExoFormer variants exhibit signs of representation collapse. We explain this via an Offloading Hypothesis: external anchors preserve essential token identity, allowing layers to specialize exclusively in computational refinement. We release codes and models to facilitate future research.

Figures (6)

• Figure 1: Overview of the proposed architectures and their training performance.
• Figure 2: Attention-pattern and representation analysis across model variants. Elementwise is used unless stated otherwise. Some graphs include input embeddings for comparison.
• Figure 3: Heatmaps showing the learned mixing coefficient ratio $\lambda_{n,1}/\lambda_{n,2}$ for each residualized component $\{Q,K,V,G\}$ across layers (y-axis) and channels/heads (x-axis). This ratio quantifies the model's reliance on the anchor relative to the current layer's projection; a higher value indicates stronger reuse of the early signal. Even with elementwise freedom, the learned pattern is largely constant within heads for $V$, while $Q$, $K$, and $G$ exhibit finer-grained structure. The dynamic variant shows more uniform distributions across channels and layers.
• Figure 4: The ratio $\lambda_{n,1}/\lambda_{n,2}$ plotted for each component ($Q$, $K$, $V$, $G$) across layers for models using elementwise mixing. Values greater than 1 indicate stronger reliance on the anchor signal, while values less than 1 indicate preference for current-layer projections.
• Figure 5: Depth-wise analysis of gating behavior across model variants.