Verbal Process Supervision Elicits Better Coding Agents

TL;DR

The paper addresses the challenge that large language models struggle with multi-step reasoning and debugging in code generation. It introduces CURA, a Code Understanding and Reasoning Agent augmented with Verbal Process Supervision (VPS) that yields step-level verbal reward signals to guide reasoning without fine-tuning. Empirical results on BigCodeBench show a 3.65% improvement over baselines, and CURA achieves state-of-the-art performance when paired with the o3-mini model and VPS; results generalize across different chat models and temperatures, with deterministic decoding often yielding the best reliability. Overall, CURA with VPS offers a tuning-free, reasoning-driven framework that enhances complex software-engineering tasks and suggests potential applicability to other reasoning-intensive domains.

Abstract

The emergence of large language models and their applications as AI agents have significantly advanced state-of-the-art code generation benchmarks, transforming modern software engineering tasks. However, even with test-time computed reasoning models, these systems still struggle with complex software engineering challenges. This work introduces CURA, a code understanding and reasoning agent system enhanced with verbal process supervision (VPS), achieving a 3.65\% improvement over baseline models on challenging benchmarks like BigCodeBench. Furthermore, CURA, when paired with the o3-mini model and VPS techniques, attains state-of-the-art performance. This work represents a step forward in integrating reasoning-driven architectures with LLM-based code generation, enabling agentic reasoning for language models to solve complex software engineering tasks.

Figures (3)

• Figure 1: The CURA architecture: a process-supervised reasoning framework incorporating verbal reward signals.
• Figure 2: Comparison of o3-mini Baseline vs. o3-mini CURA with VPS on the BigCodeBench (Hard) dataset. The y-axis shows the score (in %), while the x-axis shows three different evaluation modes (Complete, Instruct, and the Average of all modes). Notice that o3-mini VPS shows an improvement in all categories, with the largest gain in the “Complete” mode.
• Figure 3: Performance comparison of GPT-4o-mini and Mistral Large Latest on the BigCodeBench using CURA architecture with VPS technique - Hard Benchmark across different temperature settings. The models are evaluated in three categories: Complete, Instruct, and Average. Results indicate that deterministic decoding (Temp=0) generally leads to higher scores, particularly in the Complete category where Mistral Large Latest outperforms other configurations. Increasing temperature (Temp=1) negatively impacts performance across all categories, highlighting the trade-offs between deterministic and stochastic decoding in code generation tasks.