Bilevel Autoresearch: Meta-Autoresearching Itself

Abstract

If autoresearch is itself a form of research, then autoresearch can be applied to research itself. We take this idea literally: we use an autoresearch loop to optimize the autoresearch loop. Every existing autoresearch system -- from Karpathy's single-track loop to AutoResearchClaw's multi-batch extension and EvoScientist's persistent memory -- was improved by a human who read the code, identified a bottleneck, and wrote new code. We ask whether an LLM can do the same, autonomously. We present Bilevel Autoresearch, a bilevel framework where an outer loop meta-optimizes the inner autoresearch loop by generating and injecting new search mechanisms as Python code at runtime. The inner loop optimizes the task; the outer loop optimizes how the inner loop searches. Both loops use the same LLM -- no stronger model is needed at the meta level. On Karpathy's GPT pretraining benchmark, the meta-autoresearch outer loop achieves a 5x improvement over the standard inner loop alone (-0.045 vs. -0.009 val_bpb), while parameter-level adjustment without mechanism change yields no reliable gain. The outer loop autonomously discovers mechanisms from combinatorial optimization, multi-armed bandits, and design of experiments -- without human specification of which domains to explore. These mechanisms succeed by breaking the inner loop's deterministic search patterns, forcing exploration of directions the LLM's priors systematically avoid. The core principle is simple: if autoresearch can meta-autoresearch itself, it can, in principle, meta-autoresearch anything with a measurable objective.

Figures (5)

• Figure 1: Bilevel autoresearch: the inner loop optimizes the task output; the outer loop optimizes the inner loop's search mechanism by generating and injecting new Python code at runtime.
• Figure 2: Bilevel Autoresearch architecture. Level 1 (blue) runs the standard propose--train--evaluate loop. Level 1.5 (amber) adjusts search parameters every 5 iterations. Level 2 (green) generates new Python mechanisms via a 4-round research session and injects them at runtime.
• Figure 3: Level 2 research session. Each session makes four LLM calls, producing a validated Python module that modifies the inner loop's search behavior.
• Figure 4: Running-minimum val_bpb vs. iteration for all 12 runs (4 groups $\times$ 3 repeats). Thin lines: individual repeats; thick lines: group means. Groups C and D show sharp drops after Level 2 mechanisms guide the search toward TOTAL_BATCH_SIZE reduction.
• Figure 5: Excerpt from a Level 2 generated mechanism (Tabu Search Manager). This code---written entirely by the LLM during a research session---prevents the inner loop from revisiting recently explored parameter regions, breaking the deterministic proposal patterns observed in Group A.