Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution

Andrew Borthwick, Stephen Ash

TL;DR

RoboPhD demonstrates that AI agents can autonomously conduct the research cycle—hypothesis generation, experimentation, and refinement—to improve Text-to-SQL performance starting from a minimal baseline. It achieves this through a two-artifact architecture (offline database analysis tool and online SQL-generation instructions) coupled with an ELO-based evolutionary loop that cross-pollinates techniques from top performers. The system yields meaningful gains across model tiers, most notably a substantial improvement on cheaper models, and achieves 73.67% accuracy on the BIRD test set, illustrating practical deployment potential. By open-sourcing the framework and relying on minimal human domain knowledge, RoboPhD suggests a path toward scalable autonomous capability development beyond Text-to-SQL.

Abstract

We present RoboPhD, a system where AI agents autonomously conduct research to improve Text-to-SQL performance. RoboPhD implements a closed-loop evolution cycle with two coordinated components: a SQL Generation agent composed of a database analysis script and SQL generation instructions, and an Evolution agent that designs new versions based on performance feedback. Central to the framework is an ELO-based selection mechanism enabling survival-of-the-fittest dynamics while handling non-transitivity in performance. Starting from a naive 70-line baseline, RoboPhD evolves agents through iterative cross-pollination, discovering effective techniques without any external guidance on the Text-to-SQL domain. Our best agent, evolved to 1500 lines over 18 iterations, autonomously discovered strategies such as size-adaptive database analysis that adjusts depth based on schema complexity and SQL generation patterns for column selection, evidence interpretation, and aggregation. Evolution provides the largest gains on cheaper models: while we improve by 2.3 points over a strong Claude Opus 4.5 naive baseline, we show an improvement of 8.9 points over the weaker Claude Haiku model. This enables 'skip a tier' deployment: evolved Haiku exceeds naive Sonnet accuracy, and evolved Sonnet exceeds naive Opus, both at lower cost. The full system achieves 73.67% accuracy on the BIRD test set, demonstrating that AI can autonomously build a strong agentic system with only a trivial human-provided starting point.

RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution

TL;DR

RoboPhD demonstrates that AI agents can autonomously conduct the research cycle—hypothesis generation, experimentation, and refinement—to improve Text-to-SQL performance starting from a minimal baseline. It achieves this through a two-artifact architecture (offline database analysis tool and online SQL-generation instructions) coupled with an ELO-based evolutionary loop that cross-pollinates techniques from top performers. The system yields meaningful gains across model tiers, most notably a substantial improvement on cheaper models, and achieves 73.67% accuracy on the BIRD test set, illustrating practical deployment potential. By open-sourcing the framework and relying on minimal human domain knowledge, RoboPhD suggests a path toward scalable autonomous capability development beyond Text-to-SQL.

Abstract

We present RoboPhD, a system where AI agents autonomously conduct research to improve Text-to-SQL performance. RoboPhD implements a closed-loop evolution cycle with two coordinated components: a SQL Generation agent composed of a database analysis script and SQL generation instructions, and an Evolution agent that designs new versions based on performance feedback. Central to the framework is an ELO-based selection mechanism enabling survival-of-the-fittest dynamics while handling non-transitivity in performance. Starting from a naive 70-line baseline, RoboPhD evolves agents through iterative cross-pollination, discovering effective techniques without any external guidance on the Text-to-SQL domain. Our best agent, evolved to 1500 lines over 18 iterations, autonomously discovered strategies such as size-adaptive database analysis that adjusts depth based on schema complexity and SQL generation patterns for column selection, evidence interpretation, and aggregation. Evolution provides the largest gains on cheaper models: while we improve by 2.3 points over a strong Claude Opus 4.5 naive baseline, we show an improvement of 8.9 points over the weaker Claude Haiku model. This enables 'skip a tier' deployment: evolved Haiku exceeds naive Sonnet accuracy, and evolved Sonnet exceeds naive Opus, both at lower cost. The full system achieves 73.67% accuracy on the BIRD test set, demonstrating that AI can autonomously build a strong agentic system with only a trivial human-provided starting point.
Paper Structure (42 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 42 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-SQL Systems
  4. Method
  5. Conceptual Overview: The Graduate Student Metaphor
  6. Technical Implementation
  7. System Overview: Evolving an Agent with Two Components
  8. Offline Database Analysis Phase
  9. Online SQL Generation Phase
  10. Evolution: Creating New Artifacts
  11. Evaluation and ELO Ranking
  12. Experimental Protocol: Selection and Sampling
  13. Experiments
  14. Experimental Setup
  15. Main Results
  16. ...and 27 more sections

Figures (2)

  • Figure 1: RoboPhD system overview. Simple inputs (a naive agent, an evolution strategy, and training data) feed into an ELO-based evolutionary loop that produces a strong, production-ready agent for deployment.
  • Figure 2: The RoboPhD evolutionary cycle. The Database Analysis Tool (gray) is a deterministic Python script, not an LLM call, making offline analysis fast, very cheap, and reproducible.