Software Testing with Large Language Models: An Interview Study with Practitioners

TL;DR

This paper investigates how software testers use large language models (LLMs) in real-world settings to inform responsible integration into testing workflows. Through 15 semi-structured interviews analyzed with grounded theory, the authors identify an iterative, human-in-the-loop process for leveraging LLMs in tasks such as test case design, automation, and documentation. They synthesize a practitioner-informed five-step guideline—define objective and structure prompts; apply prompt engineering; iterate prompts; evaluate and integrate outputs; and continuous learning—highlighting the need for oversight due to hallucinations, privacy concerns, and reliability issues. The study provides a practical starting point for structuring LLM-assisted testing and motivates further research on socio-technical factors, tool support, and domain-specific practices.

Abstract

\textit{Background:} The use of large language models in software testing is growing fast as they support numerous tasks, from test case generation to automation, and documentation. However, their adoption often relies on informal experimentation rather than structured guidance. \textit{Aims:} This study investigates how software testing professionals use LLMs in practice to propose a preliminary, practitioner-informed guideline to support their integration into testing workflows. \textit{Method:} We conducted a qualitative study with 15 software testers from diverse roles and domains. Data were collected through semi-structured interviews and analyzed using grounded theory-based processes focused on thematic analysis. \textit{Results:} Testers described an iterative and reflective process that included defining testing objectives, applying prompt engineering strategies, refining prompts, evaluating outputs, and learning over time. They emphasized the need for human oversight and careful validation, especially due to known limitations of LLMs such as hallucinations and inconsistent reasoning. \textit{Conclusions:} LLM adoption in software testing is growing, but remains shaped by evolving practices and caution around risks. This study offers a starting point for structuring LLM use in testing contexts and invites future research to refine these practices across teams, tools, and tasks.

Figures (1)

• Figure 1: LLM into Software Testing