Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistency Meets Verification: Enhancing Test Generation Quality in Large Language Models Without Ground-Truth Solutions

Hamed Taherkhani, Alireza DaghighFarsoodeh, Mohammad Chowdhury, Hung Viet Pham, Hadi Hemmati

TL;DR

ConVerTest tackles the problem of generating valid, reliable tests from LLMs without ground-truth implementations by combining Self-Consistency for convergent test generation, Chain-of-Verification for iterative code refinement, and a Dual Execution Agreement for consensus-based verification. The two-stage pipeline yields higher test validity ($VR$ up to 39% gains), improved line coverage ($LC$ up to 28%), and better mutation scores ($MS$ up to 18%) across BigCodeBench and LBPP benchmarks, compared to holistic and two-stage baselines. Ablation studies confirm that each component—SC, CoVe, and the two-stage process—contributes meaningfully to overall performance, while a qualitative error analysis underscores the value of cross-checking tests and code to mitigate hallucinations. The work demonstrates practical gains in reliability for autonomous software testing agents and provides data and code to support replication and future extensions in code generation and testing tasks.

Abstract

Large Language Models (LLMs) have significantly advanced automated test generation, yet existing methods often rely on ground-truth code for verification, risking bug propagation and limiting applicability in test-driven development. We present ConVerTest, a novel two-stage pipeline for synthesizing reliable tests without requiring prior code implementations. ConVerTest integrates three core strategies: (i) Self-Consistency(SC) to generate convergent test cases via majority voting; (ii) Chain-of-Verification (CoVe) for iterative, reasoning-guided code refinement; and (iii) a Dual Execution Agreement to crossvalidate code and tests through consensus. Experiments on BIGCODEBENCH and LESS BASIC PYTHON PROBLEMS (LBPP) benchmarks demonstrate that ConVerTest improves test validity, line coverage, and mutation scores by up to 39%, 28%, and 18% respectively over baselines. Our findings highlight ConVerTest as a robust solution for mitigating hallucinations and enhancing the reliability of autonomous software testing agents.

Consistency Meets Verification: Enhancing Test Generation Quality in Large Language Models Without Ground-Truth Solutions

TL;DR

ConVerTest tackles the problem of generating valid, reliable tests from LLMs without ground-truth implementations by combining Self-Consistency for convergent test generation, Chain-of-Verification for iterative code refinement, and a Dual Execution Agreement for consensus-based verification. The two-stage pipeline yields higher test validity ( up to 39% gains), improved line coverage ( up to 28%), and better mutation scores ( up to 18%) across BigCodeBench and LBPP benchmarks, compared to holistic and two-stage baselines. Ablation studies confirm that each component—SC, CoVe, and the two-stage process—contributes meaningfully to overall performance, while a qualitative error analysis underscores the value of cross-checking tests and code to mitigate hallucinations. The work demonstrates practical gains in reliability for autonomous software testing agents and provides data and code to support replication and future extensions in code generation and testing tasks.

Abstract

Large Language Models (LLMs) have significantly advanced automated test generation, yet existing methods often rely on ground-truth code for verification, risking bug propagation and limiting applicability in test-driven development. We present ConVerTest, a novel two-stage pipeline for synthesizing reliable tests without requiring prior code implementations. ConVerTest integrates three core strategies: (i) Self-Consistency(SC) to generate convergent test cases via majority voting; (ii) Chain-of-Verification (CoVe) for iterative, reasoning-guided code refinement; and (iii) a Dual Execution Agreement to crossvalidate code and tests through consensus. Experiments on BIGCODEBENCH and LESS BASIC PYTHON PROBLEMS (LBPP) benchmarks demonstrate that ConVerTest improves test validity, line coverage, and mutation scores by up to 39%, 28%, and 18% respectively over baselines. Our findings highlight ConVerTest as a robust solution for mitigating hallucinations and enhancing the reliability of autonomous software testing agents.
Paper Structure (27 sections, 6 equations, 1 figure, 4 tables)

This paper contains 27 sections, 6 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Approach
  4. Consistency-Driven Generation Stage
  5. Test Generation via SC
  6. Code Verification via CoVe
  7. Consensus Verification Stage
  8. Experiment Design
  9. Models
  10. Datasets
  11. Evaluation Metrics
  12. Validity Rate (VR)
  13. Mutation Score (MS)
  14. Line Coverage (LC)
  15. Precision and Recall
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Overview of ConVerTest. The pipeline operates in two stages: (i) the Consistency-Driven Generation Stage, which produces verified code candidates via Chain-of-Verification and generates diverse, convergent test cases through SC; and (ii) the Consensus Verification Stage, which applies a Dual Execution Agreement to cross-validate candidate solutions against generated tests and filter invalid ones.