Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

TL;DR

The paper analyzes how layer normalization placement—Post-LN, Pre-LN, and a proposed Peri-LN—shapes activation variance and gradient flow in large-scale Transformers. Through analytical bounds and extensive experiments up to 3.2B parameters, it demonstrates that Peri-LN consistently yields better variance control, steadier gradients, and greater training stability than the conventional LN placements. It also discusses practical implications for precision and hyperparameter robustness, showing Peri-LN reduces instability and improves benchmark performance, including instruction tuning. Overall, Peri-LN is presented as a unifying, effective normalization strategy that merits broader adoption in future large-scale Transformer architectures.

Abstract

Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.

Figures (39)

• Figure 1: Illustration of hidden-state variance across different model depths and training iterations. From initialization through training on $6.3$ billion tokens, we observe the growth in hidden-state variance for both Pre-LN and Post-LN architectures. The analysis is based on a $1.5$B-parameter model. Detailed settings and additional results are provided in Section \ref{['subsec:growth of hidden state']}.
• Figure 2: Placement of normalization in Transformer sub-layer.
• Figure 3: Performance comparison of Post-, Pre-, and Peri-LN Transformers during pre-training. Figure \ref{['fig:pretrain_lrwseep']} llustrates the pre-training loss across learning rates. Pre-training loss and gradient norm of best performing $400$M size Transformers are in Figure \ref{['fig:pretrain_loss']} & \ref{['fig:pretrain_gradnorm']}.
• Figure 4: Common case of early stage instability in pre-training. In most of our experiments across different random seeds, the Pre-LN architecture exhibited early-stage instability. Although we initially suspected that a high learning rate might be the root cause, lowering it did not substantially mitigate these issues. By contrast, under the same settings, Peri-LN displayed stable training curves.
• Figure 5: Final-layer gradient norms for seeds $4$ and $5$.

Theorems & Definitions (8)

• Proposition 3.1: Informal
• Proposition 2.1
• Proposition 3.1
• proof
• Proposition 3.2
• proof
• Proposition 3.3
• proof