Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation

Russ Webb, Jason Ramapuram

TL;DR

Cadmus tackles the challenge of conducting controlled program-synthesis research with small models by providing a verifiable, cost-efficient framework: an integer VM, a true-program DSL, and a transformer trained at under $200$ compute cost. In experiments, Cadmus-280M-80M-v1 demonstrates high accuracy on true-program tasks and outperforms GPT-5 on a simple DSL arithmetic task, while revealing that GPT-5 relies on unknown priors that can confound reasoning. The work highlights the value of small-scale, instrumentable models for studying induction, representations, and out-of-distribution generalization, and argues for releasing the framework to enable broader, low-cost investigations. Overall, Cadmus offers a tangible platform for curriculum design and analysis of numerical representations and generalization in program synthesis, with demonstrated $100 ext{ extpercent}$ vs $95 ext{ extpercent}$ accuracy on the target task and potential for broader exploration within a fixed $65$-instruction VM.

Abstract

What research can be pursued with small models trained to complete true programs? Typically, researchers study program synthesis via large language models (LLMs) which introduce issues such as knowing what is in or out of distribution, understanding fine-tuning effects, understanding the effects of tokenization, and higher demand on compute and storage to carry out experiments. We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under \$200 of compute cost. The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution and the ability to inspect and instrument models. Smaller models working on complex reasoning tasks enable instrumentation and investigations that may be prohibitively expensive on larger models. To demonstrate that these tasks are complex enough to be of interest, we show that these Cadmus models outperform GPT-5 (by achieving 100\% accuracy while GPT-5 has 95\% accuracy) even on a simple task of completing correct, integer arithmetic programs in our domain-specific language (DSL) while providing transparency into the dataset's relationship to the problem. We also show that GPT-5 brings unknown priors into its reasoning process when solving the same tasks, demonstrating a confounding factor that prevents the use of large-scale LLMs for some investigations where the training set relationship to the task needs to be fully understood.

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation

TL;DR

Cadmus tackles the challenge of conducting controlled program-synthesis research with small models by providing a verifiable, cost-efficient framework: an integer VM, a true-program DSL, and a transformer trained at under compute cost. In experiments, Cadmus-280M-80M-v1 demonstrates high accuracy on true-program tasks and outperforms GPT-5 on a simple DSL arithmetic task, while revealing that GPT-5 relies on unknown priors that can confound reasoning. The work highlights the value of small-scale, instrumentable models for studying induction, representations, and out-of-distribution generalization, and argues for releasing the framework to enable broader, low-cost investigations. Overall, Cadmus offers a tangible platform for curriculum design and analysis of numerical representations and generalization in program synthesis, with demonstrated vs accuracy on the target task and potential for broader exploration within a fixed -instruction VM.

Abstract

What research can be pursued with small models trained to complete true programs? Typically, researchers study program synthesis via large language models (LLMs) which introduce issues such as knowing what is in or out of distribution, understanding fine-tuning effects, understanding the effects of tokenization, and higher demand on compute and storage to carry out experiments. We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under \$200 of compute cost. The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution and the ability to inspect and instrument models. Smaller models working on complex reasoning tasks enable instrumentation and investigations that may be prohibitively expensive on larger models. To demonstrate that these tasks are complex enough to be of interest, we show that these Cadmus models outperform GPT-5 (by achieving 100\% accuracy while GPT-5 has 95\% accuracy) even on a simple task of completing correct, integer arithmetic programs in our domain-specific language (DSL) while providing transparency into the dataset's relationship to the problem. We also show that GPT-5 brings unknown priors into its reasoning process when solving the same tasks, demonstrating a confounding factor that prevents the use of large-scale LLMs for some investigations where the training set relationship to the task needs to be fully understood.
Paper Structure (7 sections, 3 figures, 3 tables)

This paper contains 7 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction and Related Work
  2. Experimental Results
  3. Comparing Different Program Producing the Same Values
  4. Building Representations of Computed Values
  5. Accuracy vs. Maximum Output Tokens
  6. Conclusion
  7. Acknowledgements

Figures (3)

  • Figure 1: The accuracy of predicting the correct comparison X and Y values is shown for in and out of distribution for the Cadmus-280M-80M-v1 model. The domain in-distribution for Cadmus-280M-80M-v1 is shown for GPT-5 models when given the instructions shown the Appendix A and 2k output tokens. When provided with correct instruction describing the instructions in the alternate form, the GPT models are not able to provide useful responses using 2k output tokens (details in Appendix Figure \ref{['fig:acc_vs_tokens']}). In all cases, the accuracies reported are obtained by testing 10 validation programs per pixel (i.e. for each X,Y value pair).
  • Figure 2: The accuracies of logistic regression prediction of the two numbers computed during the number comparison task from the final transformer block representations at each token is shown. Tokens 1--5 are computing the first number (blue), while the 6--10 tokens compute the second number; finally, token 11 is the comparison instruction. The dashed lines show the accuracy obtained by guessing the majority answer in the dataset given the number of instructions seen.
  • Figure 3: The accuracy of predicting the correct comparison X and Y values is shown while varying the maximum output tokens (the Cadmus model produces an answer for each sample via predicting the next token).