Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
Anastasiia Grishina, Vadim Liventsev, Aki Härmä, Leon Moonen
TL;DR
This work addresses the near-miss syndrome in LLM-driven program synthesis by proposing SEIDR, a five-agent autonomous loop that combines Synthesize, Execute, Instruct, Debug, and Rank. The approach casts program synthesis as an iterative repair-and-replace search, guided by bug explanations and repair instructions produced by instruction-tuned LLMs, with ranking strategies such as tournament and lexicase. Across PSB2 and HumanEval-X, SEIDR with Codex, GPT-3.5, and Llama 3 demonstrates substantial gains over non-iterative baselines, solving up to 163/164 HumanEval-C++ tasks across restarts and achieving high pass rates (e.g., mean pass@100 of 84.2% with Llama 3-8B on HumanEval-C++). The work shows that SEIDR markedly improves autonomous programming efficiency, requiring far fewer executions than traditional genetic programming, and provides a robust, generalizable framework for code repair using instruction-tuned LLMs. The replication package and open science commitments further enhance practical impact for SEIDR applications in software engineering tasks.
Abstract
Program synthesis with Large Language Models (LLMs) suffers from a "near-miss syndrome": the generated code closely resembles a correct solution but fails unit tests due to minor errors. We address this with a multi-agent framework called Synthesize, Execute, Instruct, Debug, and Repair (SEIDR). Effectively applying SEIDR to instruction-tuned LLMs requires determining (a) optimal prompts for LLMs, (b) what ranking algorithm selects the best programs in debugging rounds, and (c) balancing the repair of unsuccessful programs with the generation of new ones. We empirically explore these trade-offs by comparing replace-focused, repair-focused, and hybrid debug strategies. We also evaluate lexicase and tournament selection to rank candidates in each generation. On Program Synthesis Benchmark 2 (PSB2), our framework outperforms both conventional use of OpenAI Codex without a repair phase and traditional genetic programming approaches. SEIDR outperforms the use of an LLM alone, solving 18 problems in C++ and 20 in Python on PSB2 at least once across experiments. To assess generalizability, we employ GPT-3.5 and Llama 3 on the PSB2 and HumanEval-X benchmarks. Although SEIDR with these models does not surpass current state-of-the-art methods on the Python benchmarks, the results on HumanEval-C++ are promising. SEIDR with Llama 3-8B achieves an average pass@100 of 84.2%. Across all SEIDR runs, 163 of 164 problems are solved at least once with GPT-3.5 in HumanEval-C++, and 162 of 164 with the smaller Llama 3-8B. We conclude that SEIDR effectively overcomes the near-miss syndrome in program synthesis with LLMs.