Hallucination to Consensus: Multi-Agent LLMs for End-to-End Test Generation

TL;DR

CANDOR tackles automated end-to-end Java unit-test generation with off-the-shelf LLMs by coordinating multiple specialized agents in three stages: initialization, test-prefix generation, and oracle fixing. A panel-discussion mechanism mitigates LLM hallucinations, while a dual-LLM pipeline keeps oracle evaluations concise and reliable. Empirical results show CANDOR achieves comparable test-prefix quality to EvoSuite, superior mutation scores, and significantly better oracle correctness than state-of-the-art fine-tuned approaches like TOGLL, all without fine-tuning or external tooling. Ablation studies confirm the essential roles of the Planner and panel-based oracle fixing in producing high-quality tests, highlighting CANDOR’s potential for robust, adaptable Java testing workflows.

Abstract

Unit testing plays a critical role in ensuring software correctness. However, writing unit tests manually is labor-intensive, especially for strongly typed languages like Java, motivating the need for automated approaches. Traditional methods primarily rely on search-based or randomized algorithms to achieve high code coverage and produce regression oracles, which are derived from the program's current behavior rather than its intended functionality. Recent advances in LLMs have enabled oracle generation from natural language descriptions, aligning better with user requirements. However, existing LLM-based methods often require fine-tuning or rely on external tools such as EvoSuite for test prefix generation, making them costly or cumbersome to apply in practice. In this work, we propose CANDOR, a novel prompt engineering-based LLM framework for automated unit test generation in Java. CANDOR orchestrates multiple specialized LLM agents to collaboratively generate complete tests. To mitigate the notorious hallucinations in LLMs and improve oracle correctness, we introduce a novel strategy that engages multiple reasoning LLMs in a panel discussion and generates accurate oracles based on consensus. Additionally, to reduce the verbosity of reasoning LLMs' outputs, we propose a novel dual-LLM pipeline to produce concise and structured oracle evaluations. Our experiments show that CANDOR is comparable with EvoSuite in generating tests with high code coverage and clearly superior in terms of mutation score. Moreover, our prompt engineering-based approach CANDOR significantly outperforms the SOTA fine-tuning-based oracle generator TOGLL by at least 21.1 percentage points in oracle correctness on both correct and faulty source code. Further ablation studies confirm the critical contributions of key agents in generating high-quality tests.

Figures (4)

• Figure 1: Running example from the HumanEvalJava dataset
• Figure 2: Overview of CANDOR and examples. "Desp.", "Req." and "Spec." are short for "Description", "Requirement", and "Specification", respectively.
• Figure 3: Profiles of LLM agents. "Basic LLM" and "Reasoning LLM" refer to LLMs without and with reasoning capability, respectively. Each profile consists of a system prompt and a user prompt, which define the agent's role and task, respectively. Fields enclosed in double curly braces (e.g., {{...}}) in the user prompt represent variables that are dynamically filled for each specific case. The field format_instructions is included in all user prompts and specifies the expected output format for the agent. Below each profile, an example output is provided based on the running example. The Step I, II and III of CANDOR involve agent ①,②$\sim$④ and ⑤$\sim$⑨, respectively.
• Figure 4: RQ2 results: Comparison of CANDOR, TOGLL and LLM-Empirical on correctly-implemented and faulty source code in terms of oracle correctness.