Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model

Severin Primbs, Benedikt Fein, Gordon Fraser

TL;DR

AsserT5 addresses the need for intelligent assistance in selecting test assertions for existing unit tests by fine-tuning CodeT5 on a large Java-focused dataset. The approach leverages token abstraction and focal-method context to improve the precision of generated assertions, achieving up to 59.5% exact-match accuracy in controlled evaluations and strong syntactic correctness. However, when applied to real-world fault-detection tasks using Defects4J, the generated assertions detect only a minority of bugs, underscoring gaps between machine-predicted correctness and practical fault discovery. The work provides detailed datasets, preprocessing pipelines, and model configurations, demonstrating substantial performance gains over previous assertion-generation methods while highlighting the need for improved focal-method detection and integration into developer workflows for real-world impact.

Abstract

Writing good software tests can be challenging, therefore approaches that support developers are desirable. While generating complete tests automatically is such an approach commonly proposed in research, developers may already have specific test scenarios in mind and thus just require help in selecting the most suitable test assertions for these scenarios. This can be done using deep learning models to predict assertions for given test code. Prior research on assertion generation trained these models specifically for the task, raising the question how much the use of larger models pre-trained on code that have emerged since then can improve their performance. In particular, while abstracting identifiers has been shown to improve specifically trained models, it remains unclear whether this also generalises to models pre-trained on non-abstracted code. Finally, even though prior work demonstrated high accuracy it remains unclear how this translates into the effectiveness of the assertions at their intended application -- finding faults. To shed light on these open questions, in this paper we propose AsserT5, a new model based on the pre-trained CodeT5 model, and use this to empirically study assertion generation. We find that the abstraction and the inclusion of the focal method are useful also for a fine-tuned pre-trained model, resulting in test assertions that match the ground truth assertions precisely in up to 59.5\% of cases, more than twice as precise as prior models. However, evaluation on real bugs from the Defects4J dataset shows that out of 138 bugs detectable with assertions in real-world projects, AsserT5 was only able to suggest fault-finding assertions for 33, indicating the need for further improvements.

AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model

TL;DR

AsserT5 addresses the need for intelligent assistance in selecting test assertions for existing unit tests by fine-tuning CodeT5 on a large Java-focused dataset. The approach leverages token abstraction and focal-method context to improve the precision of generated assertions, achieving up to 59.5% exact-match accuracy in controlled evaluations and strong syntactic correctness. However, when applied to real-world fault-detection tasks using Defects4J, the generated assertions detect only a minority of bugs, underscoring gaps between machine-predicted correctness and practical fault discovery. The work provides detailed datasets, preprocessing pipelines, and model configurations, demonstrating substantial performance gains over previous assertion-generation methods while highlighting the need for improved focal-method detection and integration into developer workflows for real-world impact.

Abstract

Writing good software tests can be challenging, therefore approaches that support developers are desirable. While generating complete tests automatically is such an approach commonly proposed in research, developers may already have specific test scenarios in mind and thus just require help in selecting the most suitable test assertions for these scenarios. This can be done using deep learning models to predict assertions for given test code. Prior research on assertion generation trained these models specifically for the task, raising the question how much the use of larger models pre-trained on code that have emerged since then can improve their performance. In particular, while abstracting identifiers has been shown to improve specifically trained models, it remains unclear whether this also generalises to models pre-trained on non-abstracted code. Finally, even though prior work demonstrated high accuracy it remains unclear how this translates into the effectiveness of the assertions at their intended application -- finding faults. To shed light on these open questions, in this paper we propose AsserT5, a new model based on the pre-trained CodeT5 model, and use this to empirically study assertion generation. We find that the abstraction and the inclusion of the focal method are useful also for a fine-tuned pre-trained model, resulting in test assertions that match the ground truth assertions precisely in up to 59.5\% of cases, more than twice as precise as prior models. However, evaluation on real bugs from the Defects4J dataset shows that out of 138 bugs detectable with assertions in real-world projects, AsserT5 was only able to suggest fault-finding assertions for 33, indicating the need for further improvements.

Paper Structure

This paper contains 28 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Deep-Learning-Based Assertion Generation
  4. T5 Transformer Models
  5. AsserT5
  6. Base Dataset: Methods2test
  7. Training and Evaluation Dataset
  8. Data Filtering
  9. Data Preprocessing
  10. Model
  11. Evaluation
  12. RQ1: Relevance of Context
  13. Experimental Setup
  14. Threats to Validity
  15. Results
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Example Java method and its unit test when given to a deep-learning-based assertion generation model that is trained to replace the <ASSERTION> placeholder.
  • Figure 2: Abstraction of the token sequence from \ref{['fig:example-assertion-generation']}.
  • Figure 3: System message $S$ and prompt template $P$ for with placeholders for focal and test methods.
  • Figure 4: Comparison of the individual models. score and accuracy compare the full assertion statement. The precision and recall scores only consider the assertion type, the JUnit assertion method (see \ref{['tab:considered-assertion-types']}).