Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis

Xiujing Guo, Chen Li, Tatsuhiro Tsuchiya

TL;DR

The paper addresses boundary value test input generation for white-box software testing and proposes a framework that leverages prompt-engineered LLMs to generate boundary-focused inputs. It systematically compares LLM-generated inputs against traditional methods (Random Testing, MLBVA) and concolic testing (Klee), using fault-detection rates and code-coverage metrics derived from Gcov across eight programs. Findings show that LLMs can match or exceed baseline performance in some cases, with a notable positive correlation between branch coverage and fault detection ($R^2$) for boundary-focused inputs, though gains are not universal and increasing input counts yields diminishing returns. The work suggests practical guidance for prompt design and hybrid strategies that combine LLM-based generation with established techniques to advance automated boundary value testing.

Abstract

As software systems grow more complex, automated testing has become essential to ensuring reliability and performance. Traditional methods for boundary value test input generation can be time-consuming and may struggle to address all potential error cases effectively, especially in systems with intricate or highly variable boundaries. This paper presents a framework for assessing the effectiveness of large language models (LLMs) in generating boundary value test inputs for white-box software testing by examining their potential through prompt engineering. Specifically, we evaluate the effectiveness of LLM-based test input generation by analyzing fault detection rates and test coverage, comparing these LLM-generated test sets with those produced using traditional boundary value analysis methods. Our analysis shows the strengths and limitations of LLMs in boundary value generation, particularly in detecting common boundary-related issues. However, they still face challenges in certain areas, especially when handling complex or less common test inputs. This research provides insights into the role of LLMs in boundary value testing, underscoring both their potential and areas for improvement in automated testing methods.

Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis

TL;DR

The paper addresses boundary value test input generation for white-box software testing and proposes a framework that leverages prompt-engineered LLMs to generate boundary-focused inputs. It systematically compares LLM-generated inputs against traditional methods (Random Testing, MLBVA) and concolic testing (Klee), using fault-detection rates and code-coverage metrics derived from Gcov across eight programs. Findings show that LLMs can match or exceed baseline performance in some cases, with a notable positive correlation between branch coverage and fault detection () for boundary-focused inputs, though gains are not universal and increasing input counts yields diminishing returns. The work suggests practical guidance for prompt design and hybrid strategies that combine LLM-based generation with established techniques to advance automated boundary value testing.

Abstract

As software systems grow more complex, automated testing has become essential to ensuring reliability and performance. Traditional methods for boundary value test input generation can be time-consuming and may struggle to address all potential error cases effectively, especially in systems with intricate or highly variable boundaries. This paper presents a framework for assessing the effectiveness of large language models (LLMs) in generating boundary value test inputs for white-box software testing by examining their potential through prompt engineering. Specifically, we evaluate the effectiveness of LLM-based test input generation by analyzing fault detection rates and test coverage, comparing these LLM-generated test sets with those produced using traditional boundary value analysis methods. Our analysis shows the strengths and limitations of LLMs in boundary value generation, particularly in detecting common boundary-related issues. However, they still face challenges in certain areas, especially when handling complex or less common test inputs. This research provides insights into the role of LLMs in boundary value testing, underscoring both their potential and areas for improvement in automated testing methods.
Paper Structure (15 sections, 4 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Test Input Generation Methods
  4. LLMs in Software Testing
  5. Challenges and Opportunities
  6. Methodology
  7. Framework Overview for Evaluating Boundary Input Generation Using LLMs
  8. Prompt Engineering
  9. Fault Detection Ability
  10. Algorithm
  11. Experiments
  12. Baseline Test Input Generation Methods
  13. Experimental Setup
  14. Result and Discussion
  15. Conclusion

Figures (4)

  • Figure 1: Framework for evaluating LLM-based boundary test input generation using prompt engineering.
  • Figure 2: Comparison of kill rates, statement and branch coverage for boundary and general test inputs across four prompts.
  • Figure 5: Analysis of the relationship between input count, kill rate and coverage rate.
  • Figure 6: $R^2$ correlation coefficient between branch coverage and fault detection rate across different programs.