A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability

TL;DR

This work tackles the lack of standardized benchmarks for evaluating large language models (LLMs) in automated unit-test generation. It introduces GBCV, Generated Benchmark from Control-Flow Structure and Variable Usage Composition, a CFG/data-flow-guided method to generate scalable program benchmarks that span simple to complex control-flow structures. The study evaluates GPT-3-Turbo, GPT-4o-mini, and GPT-4o on these benchmarks, finding GPT-4o excels with complex program structures while boundary-value detection remains reliable in simple conditions; arithmetic computations, however, remain a persistent challenge. The results establish key strengths and limitations of current LLMs in test generation and provide a flexible framework for expanding benchmarks and guiding future model improvements.

Abstract

Software testing ensures the quality and reliability of software products, but manual test case creation is labor-intensive. With the rise of large language models (LLMs), there is growing interest in unit test creation with LLMs. However, effective assessment of LLM-generated test cases is limited by the lack of standardized benchmarks that comprehensively cover diverse programming scenarios. To address the assessment of LLM's test case generation ability and lacking dataset for evaluation, we propose the Generated Benchmark from Control-Flow Structure and Variable Usage Composition (GBCV) approach, which systematically generates programs used for evaluating LLMs' test generation capabilities. By leveraging basic control-flow structures and variable usage, GBCV provides a flexible framework to create a spectrum of programs ranging from simple to complex. Because GPT-4o and GPT-3-Turbo are publicly accessible models, to present real-world regular user's use case, we use GBCV to assess LLM performance on them. Our findings indicate that GPT-4o performs better on complex program structures, while all models effectively detect boundary values in simple conditions but face challenges with arithmetic computations. This study highlights the strengths and limitations of LLMs in test generation, provides a benchmark framework, and suggests directions for future improvement.

Figures (9)

• Figure 1: The process of the GBCV approach
• Figure 2: An example to describe the steps of the Python program generation
• Figure 3: Simple program structure for code generation. The colors point to placeholder nodes that are replaced with the statements from corresponding sets of statement candidates. Examples include x def (x=15), x def with x c-use (x=x+10), y def (y = 7), and y-def with y c-use (y = y + 7) in Sequence. These example statements are used to replace the respective nodes.
• Figure 4: Composite program structure for code generation. The color nodes indicate placeholders that are replaced with statements selected from the corresponding candidate sets. In addition to x def, y def, x c-use, and y c-use described earlier, examples of x p-use and y p-use include x < 5 and y > 10, respectively. Compound predicates consider union (or) and intersection (and) in the logical operations.
• Figure 5: Examples of incomplete test cases - program that do not provide expected output values in the response for all three LLMs