QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

TL;DR

QualityFlow introduces an agentic workflow for program synthesis that couples a Code Generator, Test Designer, and Self-Debugger with a central Quality Checker controller. The Quality Checker uses Imagined Execution to simulate and verify unit-test conformity, and can invoke a Problem Clarifier or Revert to manage workflow trajectories, while a Test Quality Checker filters out faulty tests to prevent misleading debugging. Diversified Prompting further boosts the chances of producing a correct solution by running multiple prompts in parallel and using post-hoc selection. Across MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus, QualityFlow achieves state-of-the-art pass@1 results, with final MBPP around 94.2% and HumanEval around 98.8%, demonstrating the effectiveness of explicit quality control in guiding multi-agent code synthesis.

Abstract

We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.

Figures (5)

• Figure 1: An example problem from the MBPP code generation benchmark (No. 11). How does the model know that the generated code will pass the unit tests? Existing code generation systems assume this implicitly.
• Figure 2: In Quality Flow, the Quality Checker checks the intermediate code at every level. If the code passes the quality check, then it is accepted as the final solution; otherwise, next agents perform more processing.
• Figure 3: The pass@1 accuracy versus the workflow steps for the standard QualityFlow and the QualityFlow without Code Quality Checker (CQC) with Sonnet LLM. The CQC improves the overall pass@1 performance of the workflow and all workflow steps.
• Figure 4: The pass@1 performance of QualityFlow when the Test Quality Checker (TQC) is removed. Test Quality Checker improves the pass@1 accuracy by 1% on MBPP compared to the standard setting. On HumanEval, removing TQC causes self-debugging performance to drop, but the drop is rectified by Clarifier later.
• Figure : QualityFlow