An LLM-based Readability Measurement for Unit Tests' Context-aware Inputs

TL;DR

This work tackles the readability bottleneck of automated unit tests by introducing Context Consistency Criterion (C3), which mines readability contexts for primitive-type test inputs from the tested source code using LLMs and then judges inputs for context-consistency. It also extends EvoSuite with C3 (EvoSuiteC3) to bias test generation toward readable string inputs and appropriate number formats. Across 107 Java projects and 409 classes, C3 demonstrates strong context-mining accuracy (Precision ~84%, Recall ~83%, F1 ~83.7%), and EvoSuiteC3 significantly improves string-input readability (up to ~90%) while maintaining high context coverage. Comparisons with manual tests, EvoSuite, Randoop, and ChatUniTest show that LLM-based and context-aware approaches substantially outperform traditional tools for readability, though some variability remains due to LLM randomness and context coverage limits. A programmer study indicates that when C3 detects readable differences, participants tend to agree with its judgments, underscoring C3’s practical relevance for producing maintainable, regression-test inputs.

Abstract

Automated test techniques usually generate unit tests with higher code coverage than manual tests. However, the readability of automated tests is crucial for code comprehension and maintenance. The readability of unit tests involves many aspects. In this paper, we focus on test inputs. The central limitation of existing studies on input readability is that they focus on test codes alone without taking the tested source codes into consideration, making them either ignore different source codes' different readability requirements or require manual efforts to write readable inputs. However, we observe that the source codes specify the contexts that test inputs must satisfy. Based on such observation, we introduce the \underline{C}ontext \underline{C}onsistency \underline{C}riterion (a.k.a, C3), which is a readability measurement tool that leverages Large Language Models to extract primitive-type (including string-type) parameters' readability contexts from the source codes and checks whether test inputs are consistent with those contexts. We have also proposed EvoSuiteC3. It leverages C3's extracted contexts to help EvoSuite generate readable test inputs. We have evaluated C3's performance on $409$ \java{} classes and compared manual and automated tests' readability under C3 measurement. The results are two-fold. First, The Precision, Recall, and F1-Score of C3's mined readability contexts are \precision{}, \recall{}, and \fone{}, respectively. Second, under C3's measurement, the string-type input readability scores of EvoSuiteC3, ChatUniTest (an LLM-based test generation tool), manual tests, and two traditional tools (EvoSuite and Randoop) are $90\%$, $83\%$, $68\%$, $8\%$, and $8\%$, showing the traditional tools' inability in generating readable string-type inputs.

Figures (8)

• Figure 1: Overview of test inputs from manual and automated tests for $115$ parameters from $30$ JAVA classes (see Sec. \ref{['sec:eval:compare']}): Manual, EvoSuiteC3 (see Sec. \ref{['fitness:c3']}), and ChatUniTest xie2023chatunitest tests contain many real-world entities, such as person names, emails, URLs, and locations; EvoSuite FraserEvoSuite and Randoop pacheco2007randoop tests contain many random inputs, as well as JAVA class names, JSON strings, and UUIDs JavaUUID since these two tools tend to use predefined seeds and static or dynamic information of classes under test as test inputs
• Figure 2: Overview of Context Consistency Criterion (C3)
• Figure 3: An example of context mining prompts for LLM
• Figure 4: An example of context matching prompts for LLM
• Figure 5: Overview of EvoSuiteC3