Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training

Jonathan Lys, Vincent Gripon, Bastien Pasdeloup, Lukas Mauch, Fabien Cardinaux, Ghouthi Boukli Hacene

TL;DR

The paper reframes Transformer depth as iterative refinement of a shared latent state and asks whether extra refinement can be gained at inference without training. It introduces inference-time inner looping, applying a chosen block range repeatedly and stabilizing it with representation interpolation (regularized looping). The authors formalize naive and regularized looping, propose three regularizers, and demonstrate modest but consistent gains on several benchmarks, with latent trajectories showing smooth, small-scale shifts consistent with continued semantic refinement. The work provides a lightweight, training-free method to expose latent computation in frozen Transformers and supports the logits refinement view of depth.

Abstract

Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block. As a consequence, the inner representations (i.e. the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation. Under this lens, many works suggest that the inner space is shared across layers, meaning that tokens can be decoded at early stages. Mechanistic interpretability even goes further by conjecturing that some layers act as refinement layers. Following this path, we propose inference-time inner looping, which prolongs refinement in pretrained off-the-shelf language models by repeatedly re-applying a selected block range. Across multiple benchmarks, inner looping yields modest but consistent accuracy improvements. Analyses of the resulting latent trajectories suggest more stable state evolution and continued semantic refinement. Overall, our results suggest that additional refinement can be obtained through simple test-time looping, extending computation in frozen pretrained models.

Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training

TL;DR

The paper reframes Transformer depth as iterative refinement of a shared latent state and asks whether extra refinement can be gained at inference without training. It introduces inference-time inner looping, applying a chosen block range repeatedly and stabilizing it with representation interpolation (regularized looping). The authors formalize naive and regularized looping, propose three regularizers, and demonstrate modest but consistent gains on several benchmarks, with latent trajectories showing smooth, small-scale shifts consistent with continued semantic refinement. The work provides a lightweight, training-free method to expose latent computation in frozen Transformers and supports the logits refinement view of depth.

Abstract

Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block. As a consequence, the inner representations (i.e. the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation. Under this lens, many works suggest that the inner space is shared across layers, meaning that tokens can be decoded at early stages. Mechanistic interpretability even goes further by conjecturing that some layers act as refinement layers. Following this path, we propose inference-time inner looping, which prolongs refinement in pretrained off-the-shelf language models by repeatedly re-applying a selected block range. Across multiple benchmarks, inner looping yields modest but consistent accuracy improvements. Analyses of the resulting latent trajectories suggest more stable state evolution and continued semantic refinement. Overall, our results suggest that additional refinement can be obtained through simple test-time looping, extending computation in frozen pretrained models.
Paper Structure (18 sections, 12 equations, 2 figures, 1 table)

This paper contains 18 sections, 12 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Naive Looping
  4. Regularized Looping
  5. Naive looping (special case)
  6. Uniform
  7. Moving average
  8. Auto-alignment
  9. Experimental setup
  10. Model
  11. Datasets
  12. Experimental Results
  13. Assessing the naive looping method
  14. Layer Selection (ablation)
  15. Main Results
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Heatmap of the accuracy difference relative to the baseline ($0.6875$) across start–end layer configurations under uniform interpolation.
  • Figure 2: PCA projection of latent activation trajectories across model depth.