LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zhenghao Lin, Lei Ji, Yeyun Gong, Edward Choi
TL;DR
This work investigates why Transformer decoders exhibit recency bias rather than the earlier-token bias seen with stacked causal self-attention. By analyzing the interaction of causal masking with LayerNorm, residual connections, and input embedding anisotropy, it derives conditions under which recency bias appears, notably showing that LayerNorm can induce $S_{ij}$-level recency for $i \ge j > k$ (i.e., $S_{ij}$ grows with $j$). The findings demonstrate that LayerNorm is a key driver of recency bias, that residual connections do not fully extinguish it, and that embedding anisotropy can amplify it, offering guidance for designing positional encoding strategies and improving length generalization. These insights have practical implications for improving how positional information is encoded in decoder architectures and for developing more robust length-generalization methods in Transformers $($with all math represented in $...$ notation$)$.
Abstract
Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in attention scores toward earlier tokens. However, this differs from the bias toward later tokens typically observed in Transformer decoders, known as recency bias. We address this discrepancy by analyzing the interaction between causal self-attention and other architectural components. We show that stacked causal self-attention layers combined with LayerNorm induce recency bias. Furthermore, we examine the effects of residual connections and the distribution of input token embeddings on this bias. Our results provide new theoretical insights into how positional information interacts with architectural components and suggest directions for improving positional encoding strategies.