Tiny Autoregressive Recursive Models

TL;DR

The Autoregressive TRM is proposed and a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective are proposed.

Abstract

Tiny Recursive Models (TRMs) have recently demonstrated remarkable performance on ARC-AGI, showing that very small models can compete against large foundation models through a two-step refinement mechanism that updates an internal reasoning state $z$ and the predicted output $y$. Naturally, such refinement is of interest for any predictor; it is therefore natural to wonder whether the TRM mechanism could be effectively re-adopted in autoregressive models. However, TRMs cannot be simply compared to standard models because they lack causal predictive structures and contain persistent latent states that make it difficult to isolate specific performance gains. In this paper, we propose the Autoregressive TRM and evaluate it on small autoregressive tasks. To understand its efficacy, we propose a suite of models that gradually transform a standard Transformer to a Tiny Autoregressive Recursive Model in a controlled setting that fixes the block design, token stream, and next-token objective. Across compute-matched experiments on character-level algorithmic tasks, we surprisingly find that there are some two-level refinement baselines that show strong performance. Contrary to expectations, we find no reliable performance gains from the full Autoregressive TRM architecture. These results offer potential promise for two-step refinement mechanisms more broadly but caution against investing in the autoregressive TRM-specific model as a fruitful research direction.

Figures (6)

• Figure 1: Compute placement at fixed block-pass budget. Three autoregressive decoders execute the same compute (12 decoder-block evaluations) but allocate it differently: (left) Deep Transformer: 12 untied layers, (middle) Universal Transformer: one shared block unrolled for 12 recurrent steps, with step embeddings, and (right) Tiny Recursive Model: hierarchical dual-stream refinement using multiple inner updates before each solution update. We investigate which allocation yields the best generalization per block evaluation.
• Figure 2: Compute placement architectures. (a) Single hidden-state stream with final-iterate readout. (b) Adaptive halting with weighted readout across all iterates. (c) Two-stream factorization: solution $Y$ and auxiliary $Z$ with cross-conditioning. (d) Nested hierarchy: $L$ inner refinements of $Z$ per outer update of $Y$, repeated $H$ times.
• Figure 3: Model performance across tasks. Character accuracy (length $10$) on Addition, Copy, and Reverse for three architectures: Dense Transformer, Universal Transformer (UT), and autoregressive TRM. Copy and Reverse are solved by the Dense Transformer and UT (100% accuracy), while Addition remains more difficult and separates these two models (80% vs. 66%). The autoregressive TRM performs poorly on all three tasks, reaching only 12%, 11%, and 10% accuracy, respectively.
• Figure 4: Addition: accuracy stability by output position. Character accuracy by output quartile. Dense remains high and nearly flat across positions, while several single-stream recurrent models exhibit sharp late-position collapse (Q4 $\approx$ 8--10%). Dual UT largely avoids this collapse.
• Figure 5: Addition: training bottleneck on the final character. Last-character accuracy over training. Only Dense and Dual UT reliably overcome the bottleneck; the remaining compute placements stay flat near chance.