Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

TL;DR

The paper tackles the instability and inefficiency of Pre‑Layer Normalization in very deep Transformers by introducing Bounded Hyperbolic Tanh (BHyT), a plug‑in layer that bounds inputs to a non‑saturating range and couples it with a data‑driven variance control. It adds a variance‑approximation mechanism to reduce overhead, computing exact statistics once per block and approximating subsequent variances, yielding higher throughput without compromising stability. The authors prove a finite‑depth variance bound, showing that BHyT can offer smaller variance growth than LNS under a suitable condition, and they demonstrate empirically that BHyT delivers faster training (≈15.8% faster) and higher token throughput (≈4.2% higher) than RMSNorm, while maintaining or improving downstream performance on Llama‑1B/3B across pretraining and SFT benchmarks. In addition to strong pretraining stability, BHyT maintains robust SFT performance and inference speed, validating its practical impact for scalable, deep LLM training.

Abstract

Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning. However, Pre-LN is inefficient due to repeated statistical calculations and suffers from the curse of depth. As layers grow, the magnitude and variance of the hidden state escalate, destabilizing training. Efficiency-oriented normalization-free methods such as Dynamic Tanh (DyT) improve speed but remain fragile at depth. To jointly address stability and efficiency, we propose Bounded Hyperbolic Tanh (BHyT), a drop-in replacement for Pre-LN. BHyT couples a tanh nonlinearity with explicit, data-driven input bounding to keep activations within a non-saturating range. It prevents depth-wise growth in activation magnitude and variance and comes with a theoretical stability guarantee. For efficiency, BHyT computes exact statistics once per block and replaces a second normalization with a lightweight variance approximation, enhancing efficiency. Empirically, BHyT demonstrates improved stability and efficiency during pretraining, achieving an average of 15.8% faster training and an average of 4.2% higher token generation throughput compared to RMSNorm., while matching or surpassing its inference performance and robustness across language understanding and reasoning benchmarks. Our code is available at: https://anonymous.4open.science/r/BHyT

Figures (6)

• Figure 1: Architectural comparison of normalization strategies in Transformer blocks. RMSNorm applies normalization before each sublayer to stabilize activations but suffers from variance growth at scale. Peri-LN reinforces stability by normalizing both before and after each sublayer. DyT replaces normalization with a lightweight scaled tanh nonlinearity with learnable scalar $\alpha$. BHyT (ours) combines bounded $\tanh$ with data-driven variance control: it computes input variance once per block, approximates subsequent variance for efficiency, and explicitly constrains activations to a non-saturating range, thereby unifying stability and efficiency.
• Figure 2: Approximated and ground-truth activation variances for the second $\text{BHyT}_{\text{MLP}}$ layer across different transformer blocks in Llama-1B. The variance estimates are evaluated using 100 randomly sampled inputs from the C4 training corpus. The diagonal line represents the ideal $y = x$ reference. Deeper layers exhibit closer alignment with this reference, indicating improved approximation accuracy. Complete results for all 16 layers are reported in the Appendix \ref{['appx:var_approx']}.
• Figure 3: Layer-wise analysis of output statistics on the Llama-1B model. The top row shows the average absolute magnitude of activations, and the bottom row shows the average layer-wise variance. RMSNorm and DyT exhibit rapid growth with depth, reflecting the curse of depth and instability in deeper networks. LNS and Peri-LN suppress this escalation, and BHyT further stabilizes activation variability, yielding lower magnitude and variance than LNS.
• Figure 4: Pretraining performance and forward speed comparison. The first two figures denote loss curves for Llama-1B and Llama-3B, respectively. These show that BHyT achieves stable and competitive convergence, surpassing normalization-based baselines. Notably, compared to normalization-based baselines, BHyT reaches the same loss level fastest towards the end of training. The rightmost figure denotes the Llama-1B forward path speed comparison, highlighting that DyT is the fastest, whereas Peri-LN is the slowest. Overall, BHyT provides the best trade-off, combining reliable convergence with high training efficiency.
• Figure 5: Learning-rate robustness of DyT and BHyT with tanh-based method in Llama-1B. (a) Average per-layer variance as a function of depth under different learning rates; DyT exhibits rapidly increasing variance at larger learning rates, while BHyT maintains a much smaller variance that grows roughly linearly with depth. (b) Pretraining loss versus training tokens for DyT (left) and BHyT (right) across learning rates, where BHyT converges stably over a wide range of learning rates.

Theorems & Definitions (22)

• Proposition 3.0: Input scaling bound at the level $p$
• Remark 3.1
• Theorem 3.3: Variance approximation of attention output
• Theorem 3.4: Finite-depth variance bound of BHyT
• Proposition B.0: Input scaling bound at the level $p$
• proof
• Theorem B.1: Variance approximation of attention output
• proof
• Theorem B.1: Variance approximation of MLP layer output
• proof