Large Language Models for Software Testing Education: an Experience Report

Abstract

The rapid integration of Large Language Models (LLMs) into software engineering practice is reshaping how software testing activities are performed. LLMs are increasingly used to support software testing. Consequently, software testing education must evolve to prepare students for this new paradigm. However, while students have already begun to use LLMs in an ad hoc manner for testing tasks, there is limited empirical understanding of how such usage influences their testing behaviors, judgment, and learning outcomes. It is necessary to conduct a systematic investigation into how students learn to evaluate, control, and refine LLM-assisted testing results. This paper presents a mixed-methods, two-phase exploratory study on human-LLM collaboration in software testing education. In Phase I, we analyze classroom learning artifacts and interaction records from 15 students, together with a large-scale survey conducted in a national software testing competition (337 valid responses), to identify recurring prompt-related difficulties across testing tasks. The results reveal systematic interaction breakdowns, including missing contextual information, insufficient constraints, rigid one-shot prompting, and limited strategy-driven iteration, with automated test script generation emerging as a particularly heterogeneous and effort-intensive interaction context. Building on these findings, Phase II conducts an illustrative classroom practice that operationalizes the observed breakdowns into a lightweight, stage-aware prompt scaffold for test script generation, guiding students to explicitly articulate execution-relevant information such as environmental assumptions, interaction grounding, synchronization, and validation intent, and reporting descriptive shifts in students' testing-related articulation when interacting with LLMs.

Figures (6)

• Figure 1: Overview of the study design. Phase I combines classroom study and survey analysis to identify task-dependent LLM usage patterns and difficulties. Phase II illustrates a stage-aware instructional scaffold informed by Phase I, highlighting how LLM-supported tasks can be made pedagogically actionable.
• Figure 2: Distribution of prompt design issue categories across different software testing tasks
• Figure 3: Perceived effectiveness of LLM assistance across software testing tasks, as reported in the large-scale competition survey.
• Figure 4: Reported interaction rounds required to obtain usable test scripts
• Figure 5: Self-reported additional debugging time incurred due to issues in LLM-generated scripts.