Evaluating Large Language Models for the Generation of Unit Tests with Equivalence Partitions and Boundary Values
Martín Rodríguez, Gustavo Rossi, Alejandro Fernandez
TL;DR
The paper addresses the challenge of labor-intensive unit test creation by evaluating whether large language models can automatically generate tests using equivalence partitions and boundary values. It adopts a Java/JUnit baseline and GPT-4-0613, employing an optimized prompt template that incorporates requirements and code, and compares LLM-generated tests to manually crafted ones using both quantitative metrics (code coverage, mutation testing, test smells) and qualitative analysis. The study contributes an ad hoc set of 10 scenarios, a reusable prompt template, and a benchmark for comparing automated and manual tests, finding that GPT-generated tests achieve comparable branch coverage with some limitations related to assertions, floating-point handling, and test structure. Overall, the work demonstrates the potential of AI-assisted test generation while emphasizing the continued need for human supervision and thoughtful prompt design to ensure quality and correctness.
Abstract
The design and implementation of unit tests is a complex task many programmers neglect. This research evaluates the potential of Large Language Models (LLMs) in automatically generating test cases, comparing them with manual tests. An optimized prompt was developed, that integrates code and requirements, covering critical cases such as equivalence partitions and boundary values. The strengths and weaknesses of LLMs versus trained programmers were compared through quantitative metrics and manual qualitative analysis. The results show that the effectiveness of LLMs depends on well-designed prompts, robust implementation, and precise requirements. Although flexible and promising, LLMs still require human supervision. This work highlights the importance of manual qualitative analysis as an essential complement to automation in unit test evaluation.