Layer-wise Positional Bias in Short-Context Language Modeling

TL;DR

The paper investigates how positional bias is distributed across transformer layers during short-context next-token prediction. It introduces an attribution-based framework based on layer conductance and a sliding-window scheme to produce layer-wise positional importance profiles that are architecture-intrinsic and text-invariant. Key findings show a pronounced recency bias that grows with depth, a weaker primacy bias that diminishes in deeper layers, and an early-layer emphasis on content words with later layers losing word-type sensitivity. The results highlight the intrinsic structural role of position in autoregressive transformers and offer a principled method for analyzing layer-specific positional effects with implications for model interpretability and architecture design.

Abstract

Language models often show a preference for using information from specific positions in the input regardless of semantic relevance. While positional bias has been studied in various contexts, from attention sinks to task performance degradation in long-context settings, prior work has not established how these biases evolve across individual layers and input positions, or how they vary independent of task complexity. We introduce an attribution-based framework to analyze positional effects in short-context language modeling. Using layer conductance with a sliding-window approach, we quantify how each layer distributes importance across input positions, yielding layer-wise positional importance profiles. We find that these profiles are architecture-specific, stable across inputs, and invariant to lexical scrambling. Characterizing these profiles, we find prominent recency bias that increases with depth and subtle primacy bias that diminishes through model depth. Beyond positional structure, we also show that early layers preferentially weight content words over function words across all positions, while later layers lose this word-type differentiation.

Figures (32)

• Figure 1: Conductance framework for positional bias. We move sliding windows over input text, extract layer conductance for each word-position pair from each window, store them in tensor $\mathcal{C} \in \mathbb{R}^{L \times W \times P}$, and aggregate across words to obtain positional profiles $\bar{C}_{\ell,p}$.
• Figure 2: Layer-wise positional importance profiles across models for the Pie Man story. For each model and layer, curves show conductance scores averaged over words as a function of input position. Each subplot shows a consecutive layer pair (e.g., L1-2), with odd layers in lighter shades and even layers in darker shades for visual clarity. Y-axis ranges are adjusted per row (global max = 1.00). All layers exhibit a pronounced peak at recent positions and a secondary peak at primacy positions. See Appendix \ref{['app:profiles']} for results on other texts and the scrambled version, and Appendix \ref{['app:heatmaps']} for layer–position heatmaps (mean and variance).
• Figure 3: Cross-story consistency of positional importance profiles. Mean pairwise Pearson correlation between profiles across inputs for each layer. Correlations exceed 0.99 across all models, indicating text-invariant positional profiles.
• Figure 4: Evolution of positional bias across layers. (a) Recency fraction. (b) Primacy fraction. Patterns are averaged across all texts; recency increases monotonically with depth across all models.
• Figure 5: Position-averaged layer importance by part-of-speech (POS) category. Layer conductance is computed at the word level and averaged over positions before aggregation within POS groups.