Automated Test Suite Enhancement Using Large Language Models with Few-shot Prompting

TL;DR

The paper addresses the challenge of enhancing unit test suites by leveraging few-shot prompting of large language models with demonstrations drawn from diverse sources. It examines how demonstration origin and retrieval-based selection influence test quality, readability, and maintainability in hybrid human-AI codebases, evaluated on HumanEval and ClassEval using GPT-4o integrated into Copilot. Findings indicate that human-written examples yield the best coverage and correctness, while retrieval-based selection based on combined similarity of problem description and code consistently yields the most effective prompts. The work provides practical guidance for prompt design to augment existing test suites and demonstrates the potential of few-shot learning to improve software maintainability and evolution.

Abstract

Unit testing is essential for verifying the functional correctness of code modules (e.g., classes, methods), but manually writing unit tests is often labor-intensive and time-consuming. Unit tests generated by tools that employ traditional approaches, such as search-based software testing (SBST), lack readability, naturalness, and practical usability. LLMs have recently provided promising results and become integral to developers' daily practices. Consequently, software repositories now include a mix of human-written tests, LLM-generated tests, and those from tools employing traditional approaches such as SBST. While LLMs' zero-shot capabilities have been widely studied, their few-shot learning potential for unit test generation remains underexplored. Few-shot prompting enables LLMs to learn from examples in the prompt, and automatically retrieving such examples could enhance test suites. This paper empirically investigates how few-shot prompting with different test artifact sources, comprising human, SBST, or LLM, affects the quality of LLM-generated unit tests as program comprehension artifacts and their contribution to improving existing test suites by evaluating not only correctness and coverage but also readability, cognitive complexity, and maintainability in hybrid human-AI codebases. We conducted experiments on HumanEval and ClassEval datasets using GPT-4o, which is integrated into GitHub Copilot and widely used among developers. We also assessed retrieval-based methods for selecting relevant examples. Our results show that LLMs can generate high-quality tests via few-shot prompting, with human-written examples producing the best coverage and correctness. Additionally, selecting examples based on the combined similarity of problem description and code consistently yields the most effective few-shot prompts.