Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding on the Edge: LLM-generated Boundary Test Explanations

Sabinakhon Akbarova, Felix Dobslaw, Robert Feldt

TL;DR

This study investigates whether large language models can generate meaningful, trustworthy explanations for boundary value testing (BVT), addressing a gap where edges between input partitions require interpretability for practical use. Using a sequential mixed-methods design, 27 software professionals evaluated GPT-4.1 explanations for 20 boundary pairs across four functions, followed by six semi-structured interviews. Results show generally positive perceptions across clarity, correctness, completeness, and usefulness, with 63.5% of ratings in the 4–5 range, though a notable hallucination in one Date boundary pair reveals fragility in trust. From the findings, the authors distill a seven-item requirements checklist for future LLM-based boundary explanation tools and discuss how such explanations could integrate into testing workflows to improve actionability and reliability, while highlighting the need for adaptive depth, authoritative references, and interactive, live explanations to advance practical adoption.

Abstract

Boundary value analysis and testing (BVT) is fundamental in software quality assurance because faults tend to cluster at input extremes, yet testers often struggle to understand and justify why certain input-output pairs represent meaningful behavioral boundaries. Large Language Models (LLMs) could help by producing natural-language rationales, but their value for BVT has not been empirically assessed. We therefore conducted an exploratory study on LLM-generated boundary explanations: in a survey, twenty-seven software professionals rated GPT-4.1 explanations for twenty boundary pairs on clarity, correctness, completeness and perceived usefulness, and six of them elaborated in follow-up interviews. Overall, 63.5% of all ratings were positive (4-5 on a five-point Likert scale) compared to 17% negative (1-2), indicating general agreement but also variability in perceptions. Participants favored explanations that followed a clear structure, cited authoritative sources, and adapted their depth to the reader's expertise; they also stressed the need for actionable examples to support debugging and documentation. From these insights, we distilled a seven-item requirement checklist that defines concrete design criteria for future LLM-based boundary explanation tools. The results suggest that, with further refinement, LLM-based tools can support testing workflows by making boundary explanations more actionable and trustworthy.

Understanding on the Edge: LLM-generated Boundary Test Explanations

TL;DR

This study investigates whether large language models can generate meaningful, trustworthy explanations for boundary value testing (BVT), addressing a gap where edges between input partitions require interpretability for practical use. Using a sequential mixed-methods design, 27 software professionals evaluated GPT-4.1 explanations for 20 boundary pairs across four functions, followed by six semi-structured interviews. Results show generally positive perceptions across clarity, correctness, completeness, and usefulness, with 63.5% of ratings in the 4–5 range, though a notable hallucination in one Date boundary pair reveals fragility in trust. From the findings, the authors distill a seven-item requirements checklist for future LLM-based boundary explanation tools and discuss how such explanations could integrate into testing workflows to improve actionability and reliability, while highlighting the need for adaptive depth, authoritative references, and interactive, live explanations to advance practical adoption.

Abstract

Boundary value analysis and testing (BVT) is fundamental in software quality assurance because faults tend to cluster at input extremes, yet testers often struggle to understand and justify why certain input-output pairs represent meaningful behavioral boundaries. Large Language Models (LLMs) could help by producing natural-language rationales, but their value for BVT has not been empirically assessed. We therefore conducted an exploratory study on LLM-generated boundary explanations: in a survey, twenty-seven software professionals rated GPT-4.1 explanations for twenty boundary pairs on clarity, correctness, completeness and perceived usefulness, and six of them elaborated in follow-up interviews. Overall, 63.5% of all ratings were positive (4-5 on a five-point Likert scale) compared to 17% negative (1-2), indicating general agreement but also variability in perceptions. Participants favored explanations that followed a clear structure, cited authoritative sources, and adapted their depth to the reader's expertise; they also stressed the need for actionable examples to support debugging and documentation. From these insights, we distilled a seven-item requirement checklist that defines concrete design criteria for future LLM-based boundary explanation tools. The results suggest that, with further refinement, LLM-based tools can support testing workflows by making boundary explanations more actionable and trustworthy.
Paper Structure (29 sections, 3 figures, 1 table)

This paper contains 29 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Method
  4. Study Design and Rationale
  5. Context and Materials
  6. Objects under test
  7. Boundary pair generation
  8. Explanation generation
  9. Participants and Recruitment
  10. Survey
  11. Interview Procedure
  12. Data Analysis
  13. Quantitative (survey)
  14. Qualitative (survey comments & interviews)
  15. Results
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Example of a boundary pair shown to participants with an explanation. The LLM origin of explanations was revealed only during interviews.
  • Figure 2: Score distribution for the Date FUT, shown per boundary pair and for all pairs combined. The lower overall ratings --- particularly for correctness --- are due to a hallucination in the explanation of the first boundary pair.
  • Figure 3: Themes and corresponding codes identified through thematic analysis of the interview data.