Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports

Wendkûuni C. Ouédraogo, Laura Plein, Kader Kaboré, Andrew Habib, Jacques Klein, David Lo, Tegawendé F. Bissyandé

TL;DR

BRMiner’s combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.

Abstract

The quality of software is closely tied to the effectiveness of the tests it undergoes. Manual test writing, though crucial for bug detection, is time-consuming, which has driven significant research into automated test case generation. However, current methods often struggle to generate relevant inputs, limiting the effectiveness of the tests produced. To address this, we introduce BRMiner, a novel approach that leverages Large Language Models (LLMs) in combination with traditional techniques to extract relevant inputs from bug reports, thereby enhancing automated test generation tools. In this study, we evaluate BRMiner using the Defects4J benchmark and test generation tools such as EvoSuite and Randoop. Our results demonstrate that BRMiner achieves a Relevant Input Rate (RIR) of 60.03% and a Relevant Input Extraction Accuracy Rate (RIEAR) of 31.71%, significantly outperforming methods that rely on LLMs alone. The integration of BRMiner's input enhances EvoSuite ability to generate more effective test, leading to increased code coverage, with gains observed in branch, instruction, method, and line coverage across multiple projects. Furthermore, BRMiner facilitated the detection of 58 unique bugs, including those that were missed by traditional baseline approaches. Overall, BRMiner's combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.

Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports

TL;DR

BRMiner’s combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.

Abstract

The quality of software is closely tied to the effectiveness of the tests it undergoes. Manual test writing, though crucial for bug detection, is time-consuming, which has driven significant research into automated test case generation. However, current methods often struggle to generate relevant inputs, limiting the effectiveness of the tests produced. To address this, we introduce BRMiner, a novel approach that leverages Large Language Models (LLMs) in combination with traditional techniques to extract relevant inputs from bug reports, thereby enhancing automated test generation tools. In this study, we evaluate BRMiner using the Defects4J benchmark and test generation tools such as EvoSuite and Randoop. Our results demonstrate that BRMiner achieves a Relevant Input Rate (RIR) of 60.03% and a Relevant Input Extraction Accuracy Rate (RIEAR) of 31.71%, significantly outperforming methods that rely on LLMs alone. The integration of BRMiner's input enhances EvoSuite ability to generate more effective test, leading to increased code coverage, with gains observed in branch, instruction, method, and line coverage across multiple projects. Furthermore, BRMiner facilitated the detection of 58 unique bugs, including those that were missed by traditional baseline approaches. Overall, BRMiner's combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.
Paper Structure (38 sections, 2 equations, 7 figures, 12 tables, 3 algorithms)

This paper contains 38 sections, 2 equations, 7 figures, 12 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. DSE and EvoSuite
  4. LLM-based Test Input Generation
  5. Prompt Engineering
  6. Bug reports and relevant input
  7. Bug reports
  8. Relevant inputs
  9. BRMiner
  10. Usage scenario
  11. Approach overview
  12. Technical Challenges and Considerations
  13. Initial Steps: Parsing and Extraction
  14. Literal Extraction from Source Code and Natural Language Text
  15. Application of Extracted Inputs in Test Case Generation
  16. ...and 23 more sections

Figures (7)

  • Figure 1: Input value mentioned in the bug report for issue 3447 in the FasterXML jackson-databind library appears in a test case written by a developer after fixing the bug
  • Figure 2: Pattern of the input value mentioned in the bug report for issue 1205 in the Apache Commons Lang library appears in a test case written by a developer after fixing the bug
  • Figure 3: Input value 'secure&#9pass' mentioned in the bug report for issue 212 in the Azure AD Authentication Library
  • Figure 4: Overview of BRMiner, an automatic approach to extract potential test inputs from bug reports
  • Figure 6: Chain-of-Thought Prompt Design for Classifying Input Mentions in Bug Reports
  • ...and 2 more figures