Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-time Adaptation of Tiny Recursive Models

Ronan Killian McGovern

TL;DR

The paper tackles the problem of adapting abstract-reasoning models to ARC AGI II tasks under strict compute limits. It shows that starting from a pre-trained Tiny Recursive Model (TRM) and applying full fine-tuning yields meaningful post-training gains (up to 6.67% on semi-private tasks), outperforming embedding-only approaches and LoRA in many runs. Pre-training from scratch remains infeasible within the competition budget, while pre-trained TRMs can be efficiently adapted, though results exhibit substantial stochastic variability. The work also investigates data diversity, augmentation strategies, and latent-program-space perspectives (SLPS), highlighting the trade-offs and open questions in pre-training design for robust post-training generalization.

Abstract

Prior to the close of the 2025 ARC Prize competition, the leading open source approach - known as TRM, or Tiny Recursive Models - involved training a 7M parameter recursive neural network on augmented variants of ARC tasks. That approach scored approximately 7.8% on the public ARC AGI II evaluation set, but required a level of compute far in excess of what is allowed during the competition. This paper shows that, by starting from a tiny recursive model that has been pre-trained on public ARC tasks, one can efficiently fine-tune on competition tasks within the allowed compute limits. Specifically, a model was pre-trained on 1,280 public tasks for 700k+ optimizer steps over 48 hours on 4xH100 SXM GPUs to obtain a ~10% score on the public evaluation set. That model was then post-trained in just 12,500 gradient steps during the competition to reach a score of 6.67% on semi-private evaluation tasks. Notably, such post-training performance is achieved by full-fine tuning of the tiny model, not LoRA fine-tuning or fine-tuning of task embeddings alone.

Test-time Adaptation of Tiny Recursive Models

TL;DR

The paper tackles the problem of adapting abstract-reasoning models to ARC AGI II tasks under strict compute limits. It shows that starting from a pre-trained Tiny Recursive Model (TRM) and applying full fine-tuning yields meaningful post-training gains (up to 6.67% on semi-private tasks), outperforming embedding-only approaches and LoRA in many runs. Pre-training from scratch remains infeasible within the competition budget, while pre-trained TRMs can be efficiently adapted, though results exhibit substantial stochastic variability. The work also investigates data diversity, augmentation strategies, and latent-program-space perspectives (SLPS), highlighting the trade-offs and open questions in pre-training design for robust post-training generalization.

Abstract

Prior to the close of the 2025 ARC Prize competition, the leading open source approach - known as TRM, or Tiny Recursive Models - involved training a 7M parameter recursive neural network on augmented variants of ARC tasks. That approach scored approximately 7.8% on the public ARC AGI II evaluation set, but required a level of compute far in excess of what is allowed during the competition. This paper shows that, by starting from a tiny recursive model that has been pre-trained on public ARC tasks, one can efficiently fine-tune on competition tasks within the allowed compute limits. Specifically, a model was pre-trained on 1,280 public tasks for 700k+ optimizer steps over 48 hours on 4xH100 SXM GPUs to obtain a ~10% score on the public evaluation set. That model was then post-trained in just 12,500 gradient steps during the competition to reach a score of 6.67% on semi-private evaluation tasks. Notably, such post-training performance is achieved by full-fine tuning of the tiny model, not LoRA fine-tuning or fine-tuning of task embeddings alone.

Paper Structure

This paper contains 46 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Description and Brief History of the ARC Prize
  3. Tiny Recursive Models as a Basis for Solving ARC AGI II Tasks
  4. Fitting a Tiny Recursive Model Approach within Compute Requirements for the ARC Prize 2025 Competition
  5. Approach and Contribution
  6. Methods
  7. Pre-training
  8. Original Paper Replication - 100k epochs
  9. Extended Data for 200k Epochs
  10. Filtered Hard Data for 1M Epochs
  11. Post-training
  12. Simulating and Measuring Post-training Performance
  13. Post-training an ARC AGI I pre-trained model
  14. Post-training an ARC AGI II pre-trained model, but with 10 public eval tasks withheld as a test set
  15. Post-training an ARC AGI II pre-trained model via a formal ARC AGI 2025 Submission
  16. ...and 31 more sections

Figures (5)

  • Figure 1: ARC evaluation pass@2 and pass@1000 accuracies throughout training for the pretrain_att_arc2concept_4 replication run.
  • Figure 2: Evaluation and train exact accuracies throughout training for the pretrain_att_arc2concept_4 replication run.
  • Figure 3: ARC evaluation pass@2 for the ARC AGI I post-training sweeps logged in Trelis2025PosttrainEval2. Each curve shows the first $6$k optimisation steps for a different adaptation strategy.
  • Figure 4: Embedding cosine similarity among augmented variants of the same task during extended pre-training.
  • Figure 5: Embedding cosine similarity across base tasks for the extended pre-training runs.