LLM-ERM: Sample-Efficient Program Learning via LLM-Guided Search

TL;DR

This work tackles learning succinct algorithms from limited data by contrasting exhaustive short-program search with gradient-based methods that struggle on non-local tasks. It introduces LLM-ERM, a propose-and-verify framework that uses a pretrained reasoning LLM to generate a small set of candidate programs and selects among them via ERM on held-out data, avoiding gradient-based optimization. Theoretical analysis links the sample-efficiency-computational-efficiency trade-off to statistical-query lower bounds, showing exponential sample requirements for SGD on high-SQ-dimension classes like parity, while length-first enumeration remains sample-efficient but intractable computationally. Empirically, LLM-ERM achieves exact or near-exact recovery of target rules with just ~200 samples across parity variants, pattern matching, and primality, and generalizes to longer inputs, whereas SGD-based training overfits even with large datasets. The work suggests language-guided program synthesis as a practical path to robust, interpretable, and sample-efficient learning of succinct hypotheses beyond the reach of standard gradient-based training.

Abstract

We seek algorithms for program learning that are both sample-efficient and computationally feasible. Classical results show that targets admitting short program descriptions (e.g., with short ``python code'') can be learned with a ``small'' number of examples (scaling with the size of the code) via length-first program enumeration, but the search is exponential in description length. Consequently, Gradient-based training avoids this cost yet can require exponentially many samples on certain short-program families. To address this gap, we introduce LLM-ERM, a propose-and-verify framework that replaces exhaustive enumeration with an LLM-guided search over candidate programs while retaining ERM-style selection on held-out data. Specifically, we draw $k$ candidates with a pretrained reasoning-augmented LLM, compile and check each on the data, and return the best verified hypothesis, with no feedback, adaptivity, or gradients. Theoretically, we show that coordinate-wise online mini-batch SGD requires many samples to learn certain short programs. {\em Empirically, LLM-ERM solves tasks such as parity variants, pattern matching, and primality testing with as few as 200 samples, while SGD-trained transformers overfit even with 100,000 samples}. These results indicate that language-guided program synthesis recovers much of the statistical efficiency of finite-class ERM while remaining computationally tractable, offering a practical route to learning succinct hypotheses beyond the reach of gradient-based training.

Figures (19)

• Figure 1: Side-by-side comparison of reasoning traces for two distinct learning tasks. Rules were proposed by GPT-5-Thinking in a single run until convergence. Train accuracy values shown are those the model decided to compute and explicitly include in its reasoning trace. In each subfigure the two columns, Left: sequence of proposed rules and Right: rationale for each proposal.
• Figure 2: Trade-offs between sample and computational efficiency in program learning. The proposed method (LLM-ERM) lies in the intersection.
• Figure 3: An illustration of LLM-ERM. A prompt from $S_{\mathrm{tr}}$ seeds the LLM to propose candidates, which are evaluated on train and validation sets. We track the lowest validation error and stop early when it drops below $\theta$, or otherwise after $k$ iterations.
• Figure 4: Prompt used in our LLM-ERM procedure. We run GPT-5 with this prompt for up to $k$ independent iterations, each returning only Python code for a candidate target function.
• Figure 5: LLM-ERM generalizes from 200 samples, while an SGD-trained LLM overfits. With only 200 training examples per task, LLM-ERM typically recovers the target function exactly. For each input length $n$ (x-axis), models are trained and tested independently on sequences of that length. Although training Qwen3-1.7B from scratch with SGD fits the training data at each $n$, it fails to generalize as $n$ grows. For Random 10-Parity, the 10 indices used for the parity check are fixed by seed 42, while the overall input length $n$ varies. See Fig. \ref{['fig:curves_complete']} in App. \ref{['app:sgd_llm']} for additional results.

Theorems & Definitions (17)

• Proposition 1
• Definition 1: Statistical Query Dimension 10.1145/195058.195147
• Proposition 2: Lower bound for SGD
• Remark 1: Sample and runtime complexity tradeoff
• Theorem 1: 10.1145/1968.1972; see also Cor. 2.3 of 10.5555/2621980
• Corollary 1
• proof
• Proposition 3
• proof
• Definition 2: $1$-STAT and $1$-STAT(b)