Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency

TL;DR

The paper tackles enhancing LLM-driven code generation by coupling multi-agent collaboration (process-oriented) with runtime execution information-based debugging (product-oriented) in a chained ACT+Debugger framework. It empirically evaluates 19 LLMs on HumanEval and HumanEval+ to measure functional accuracy, code rigor, and latency, comparing the chained approach against six baselines. Key findings show that debugging-based methods often outperform agentic workflows, and chaining yields the most benefit when ACT and Debugger performance gaps are small, with simple two-agent collaboration sometimes outperforming more complex configurations. The work provides practical guidance for deploying robust, efficient AI-powered code generation in real-world settings and highlights model-dependent, dataset-specific nuances that influence the effectiveness of post-training strategy combinations.

Abstract

The use of large language models (LLMs) for automated code generation has emerged as a significant focus within AI research. As these pretrained models continue to evolve, their ability to understand and generate complex code structures has opened new possibilities for automating intricate programming tasks for the sake of accurate code generation. Although contemporary foundational models demonstrate promoting results, researchers continue to explore optimal post-training strategies to enhance code quality. These include supervised fine-tuning, retrieval-augmented generation (RAG), debugging, and many others. In this paper, we combine two widely used approaches namely multi-agent collaboration and runtime execution information-based debugging, for improving code generation functionality, reliability, and practical applicability. We perform an empirical study in order to extend the evaluation of the individual strategies as well as the proposed composition of the activities of both strategies. Our study use 19 LLMs to examines the performance of individual and the proposed strategies, offering comprehensive insights into how different programming activities compositions and training paradigms influence code generation effectiveness. In particular, we implement a chained system that combines both strategies to assess their combined impact on functional accuracy, code reliability, and generation latency using two benchmark datasets commonly used for code generation. Our findings provide valuable insights for organizations seeking robust AI-driven coding solutions by guiding them in selecting models that can better adapt to complex post-training strategies, ultimately fostering the adoption of more effective and reliable code generation technologies.

Figures (9)

• Figure 1: Architecture of ACT + Debugger approach.
• Figure 2: Comparison of probability density functions for ACT and ACT+Debug samples on the HumanEval dataset.
• Figure 3: Comparison of probability density functions for Debug and ACT+Debug samples on the HumanEval dataset.
• Figure 8: Composition of the different segments explored in this paper.
• Figure 9: Pass@1 accuracy scores of 19 Models tested with 6 different approaches on the HumanEval benchmark. Green lines indicate models where AC+Debug or ACT+Debug scores surpass those of the Debugger, gray lines represent cases where the scores are equal, and orange lines denote instances where the scores are lower. The red line represents the average scores of all models. Accuracy scores are measured in percentages.