Understanding LLM-Driven Test Oracle Generation
Adam Bodicoat, Gunel Jahangirova, Valerio Terragni
TL;DR
This work tackles the oracle problem in automated testing by evaluating LLM-generated test oracles driven by natural-language prompts. It systematically investigates how input context and prompting strategies affect the quality of fault-revealing oracles using the GHRB Java bug benchmark and two LLMs, StarCoder and GPT-4o. The results show that richer context, especially including the class under test (CUT), and simple prompting (zero-shot or few-shot) outperform reasoning-based prompts, with model choice being secondary to prompting design. The findings offer practical guidance for AI-assisted testing tools and underscore the importance of promptware design in enabling reliable, fault-detecting test oracles.
Abstract
Automated unit test generation aims to improve software quality while reducing the time and effort required for creating tests manually. However, existing techniques primarily generate regression oracles that predicate on the implemented behavior of the class under test. They do not address the oracle problem: the challenge of distinguishing correct from incorrect program behavior. With the rise of Foundation Models (FMs), particularly Large Language Models (LLMs), there is a new opportunity to generate test oracles that reflect intended behavior. This positions LLMs as enablers of Promptware, where software creation and testing are driven by natural-language prompts. This paper presents an empirical study on the effectiveness of LLMs in generating test oracles that expose software failures. We investigate how different prompting strategies and levels of contextual input impact the quality of LLM-generated oracles. Our findings offer insights into the strengths and limitations of LLM-based oracle generation in the FM era, improving our understanding of their capabilities and fostering future research in this area.