Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement

Manish Bhattarai, Miguel Cordova, Minh Vu, Javier Santos, Ismael Boureima, Dan O'Malley

TL;DR

ARCS tackles the challenge of reliable, fine-tuning-free code synthesis by introducing a retrieval-augmented, budgeted synthesize-execute-repair loop over a frozen LLM. By retrieving task-relevant evidence before generation, proposing candidates, running them against tests, and repairing prompts with execution feedback, ARCS achieves monotone improvement under a fixed iteration budget and provable termination. The approach combines plan-conditioned retrieval, deterministic replay, and verification-guided repair across Small/Medium/Large tiers, yielding strong results on HumanEval (up to $87.2\%$ pass@1), TransCoder translations, and the LANL CodeBLEU benchmark with a notable improvement of +0.115. This framework offers a practical, reproducible alternative to heavy fine-tuning or tree-search methods, enabling robust code synthesis using existing LLM checkpoints with predictable latency–accuracy trade-offs.

Abstract

We present Agentic Retrieval-Augmented Code Synthesis (ARCS), a system that improves LLM-based code generation without fine-tuning. ARCS operates through a budgeted synthesize-execute-repair loop over a frozen model: it retrieves relevant code context before generation, proposes candidates, executes them against tests, and repairs based on execution feedback. This retrieval-before-generation design reduces hallucination and accelerates convergence. We formalize ARCS as a state-action process with provable guarantees on termination, monotonic improvement, and bounded cost. A tiered controller (Small/Medium/Large) trades latency for accuracy predictably. On HumanEval, ARCS achieves up to 87.2% pass@1 with Llama-3.1-405B, surpassing CodeAgent (82.3%) while using simpler control than tree-search methods. On TransCoder, it achieves >= 90% accuracy on most translation pairs. On a LANL scientific corpus, it improves CodeBLEU by +0.115 over baseline RAG. ARCS provides a practical, reproducible approach to reliable code synthesis using existing LLM checkpoints.

ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement

TL;DR

ARCS tackles the challenge of reliable, fine-tuning-free code synthesis by introducing a retrieval-augmented, budgeted synthesize-execute-repair loop over a frozen LLM. By retrieving task-relevant evidence before generation, proposing candidates, running them against tests, and repairing prompts with execution feedback, ARCS achieves monotone improvement under a fixed iteration budget and provable termination. The approach combines plan-conditioned retrieval, deterministic replay, and verification-guided repair across Small/Medium/Large tiers, yielding strong results on HumanEval (up to pass@1), TransCoder translations, and the LANL CodeBLEU benchmark with a notable improvement of +0.115. This framework offers a practical, reproducible alternative to heavy fine-tuning or tree-search methods, enabling robust code synthesis using existing LLM checkpoints with predictable latency–accuracy trade-offs.

Abstract

We present Agentic Retrieval-Augmented Code Synthesis (ARCS), a system that improves LLM-based code generation without fine-tuning. ARCS operates through a budgeted synthesize-execute-repair loop over a frozen model: it retrieves relevant code context before generation, proposes candidates, executes them against tests, and repairs based on execution feedback. This retrieval-before-generation design reduces hallucination and accelerates convergence. We formalize ARCS as a state-action process with provable guarantees on termination, monotonic improvement, and bounded cost. A tiered controller (Small/Medium/Large) trades latency for accuracy predictably. On HumanEval, ARCS achieves up to 87.2% pass@1 with Llama-3.1-405B, surpassing CodeAgent (82.3%) while using simpler control than tree-search methods. On TransCoder, it achieves >= 90% accuracy on most translation pairs. On a LANL scientific corpus, it improves CodeBLEU by +0.115 over baseline RAG. ARCS provides a practical, reproducible approach to reliable code synthesis using existing LLM checkpoints.
Paper Structure (28 sections, 4 theorems, 19 equations, 1 figure, 5 tables, 1 algorithm)

This paper contains 28 sections, 4 theorems, 19 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The ARCS Framework
  4. Overview and Core Loop
  5. Task and Notation.
  6. Iterative Framework.
  7. Similarity and Filtering.
  8. Design Guarantees.
  9. Core Components: Theory and Implementation
  10. Planner $P_\phi$.
  11. Retriever $R_\psi$.
  12. Context Renderer and Proposer.
  13. Executor $\mathcal{E}$.
  14. Repair and Refresh.
  15. Computational considerations.
  16. ...and 13 more sections

Key Result

Proposition 1

Under best-so-far tracking (Eq. equation eq:bestsofar), the success metric is non-decreasing:

Figures (1)

  • Figure 1: High-level overview of ARCS. (1) Extract and embed code snippets with metadata. (2) Use CoT planning to issue plan-conditioned retrieval queries; render retrieved evidence into the prompt. (3) Propose code, execute in a sandbox against tests, and encode execution feedback to repair iteratively.

Theorems & Definitions (8)

  • Proposition 1: Monotonic Improvement
  • proof
  • Proposition 2: Bounded Termination
  • proof
  • Proposition 3: Cost Bound
  • proof
  • Lemma 1: Tier Monotonicity
  • proof