Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intra-Layer Recurrence in Transformers for Language Modeling

Anthony Nguyen, Wenjun Lin

TL;DR

The paper addresses the computational and memory growth of deep Transformer language models by proposing Intra-Layer Recurrence (ILR), a mechanism that re-enters selected layers within a single forward pass to adjust effective depth without adding parameters. ILR is defined by a per-layer reuse map $\mathbf{R}=[r_1,\dots,r_L]$ and a modified forward computation, with gradients propagating through repeated passes via equations for $\partial \mathcal{L}/\partial h^{(l-1)}$ and $\partial \mathcal{L}/\partial \theta^{(l)}$. Empirical results on LLaMA-based decoders at small and large scales show that prioritizing early-layer recurrence yields the strongest perplexity reductions, albeit at increased compute cost, while block recurrence offers mixed benefits. The findings highlight the value of granular, layer-wise recurrence for efficient language modeling and point to future work on adaptive recurrence strategies and scalable discovery of optimal recurrence distributions.

Abstract

Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures.

Intra-Layer Recurrence in Transformers for Language Modeling

TL;DR

The paper addresses the computational and memory growth of deep Transformer language models by proposing Intra-Layer Recurrence (ILR), a mechanism that re-enters selected layers within a single forward pass to adjust effective depth without adding parameters. ILR is defined by a per-layer reuse map and a modified forward computation, with gradients propagating through repeated passes via equations for and . Empirical results on LLaMA-based decoders at small and large scales show that prioritizing early-layer recurrence yields the strongest perplexity reductions, albeit at increased compute cost, while block recurrence offers mixed benefits. The findings highlight the value of granular, layer-wise recurrence for efficient language modeling and point to future work on adaptive recurrence strategies and scalable discovery of optimal recurrence distributions.

Abstract

Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures.
Paper Structure (11 sections, 5 equations, 3 figures, 5 tables)

This paper contains 11 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Gradient w.r.t. the Input
  5. Gradient w.r.t. the Parameters
  6. Experiments and Results
  7. Results
  8. Observations and Analysis
  9. Limitations
  10. Conclusion
  11. Experimental Details

Figures (3)

  • Figure 1: Transformer architecture with intra-layer recurrence.
  • Figure 2: Depth-recurrent transformer proposed by Geiping et al. geiping2025scaling, which groups layers into three blocks and applies recurrence only to the middle block.
  • Figure 3: Diagram illustrating block recurrence from small-scale experiments. Unlike ILR, recurrence is applied across the entire stack rather than per layer.