Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Retrieval-Augmented Deep Assertion Generation via Joint Training

Quanjun Zhang, Chunrong Fang, Yi Zheng, Ruixiang Qian, Shengcheng Yu, Yuan Zhao, Jianyi Zhou, Yun Yang, Tao Zheng, Zhenyu Chen

TL;DR

AG-RAG is a retrieval-augmented automated assertion generation approach that leverages external codebases and joint training to address various technical limitations of prior work and demonstrates the promising potential of jointly fine-tuning the PLM-based retriever and generator to predict accurate assertions by incorporating external knowledge sources.

Abstract

Unit testing attempts to validate the correctness of basic units of the software system under test and has a crucial role in software development and testing. Very recent work proposes a retrieve-and-edit approach to generate unit test oracles, i.e., assertions. Despite being promising, it is still far from perfect due to some limitations, such as splitting assertion retrieval and generation into two separate components without benefiting each other. In this paper, we propose AG-RAG, a retrieval-augmented automated assertion generation approach that leverages external codebases and joint training to address various technical limitations of prior work. Inspired by the plastic surgery hypothesis, AG-RAG attempts to combine relevant unit tests and advanced pre-trained language models (PLMs) with retrieval-augmented fine-tuning. AG-RAG builds a dense retriever to search for relevant test-assert pairs (TAPs) with semantic matching and a retrieval-augmented generator to synthesize accurate assertions with the focal-test and retrieved TAPs as input. Besides, AG-RAG leverages a code-aware language model CodeT5 as the cornerstone to facilitate both assertion retrieval and generation tasks. Furthermore, the retriever is optimized in conjunction with the generator as a whole pipeline with a joint training strategy. This unified design fully adapts both components specifically for retrieving more useful TAPs, thereby generating accurate assertions. We extensively evaluate AG-RAG against six state-of-the-art AG approaches on two benchmarks and three metrics. Experimental results show that AG-RAG significantly outperforms previous AG approaches on all benchmarks and metrics, e.g., improving the most recent baseline EditAS by 20.82% and 26.98% in terms of accuracy. AG-RAG also correctly generates 1739 and 2866 unique assertions that all baselines fail to generate, 3.45X and 9.20X more than EditAS.

Improving Retrieval-Augmented Deep Assertion Generation via Joint Training

TL;DR

AG-RAG is a retrieval-augmented automated assertion generation approach that leverages external codebases and joint training to address various technical limitations of prior work and demonstrates the promising potential of jointly fine-tuning the PLM-based retriever and generator to predict accurate assertions by incorporating external knowledge sources.

Abstract

Unit testing attempts to validate the correctness of basic units of the software system under test and has a crucial role in software development and testing. Very recent work proposes a retrieve-and-edit approach to generate unit test oracles, i.e., assertions. Despite being promising, it is still far from perfect due to some limitations, such as splitting assertion retrieval and generation into two separate components without benefiting each other. In this paper, we propose AG-RAG, a retrieval-augmented automated assertion generation approach that leverages external codebases and joint training to address various technical limitations of prior work. Inspired by the plastic surgery hypothesis, AG-RAG attempts to combine relevant unit tests and advanced pre-trained language models (PLMs) with retrieval-augmented fine-tuning. AG-RAG builds a dense retriever to search for relevant test-assert pairs (TAPs) with semantic matching and a retrieval-augmented generator to synthesize accurate assertions with the focal-test and retrieved TAPs as input. Besides, AG-RAG leverages a code-aware language model CodeT5 as the cornerstone to facilitate both assertion retrieval and generation tasks. Furthermore, the retriever is optimized in conjunction with the generator as a whole pipeline with a joint training strategy. This unified design fully adapts both components specifically for retrieving more useful TAPs, thereby generating accurate assertions. We extensively evaluate AG-RAG against six state-of-the-art AG approaches on two benchmarks and three metrics. Experimental results show that AG-RAG significantly outperforms previous AG approaches on all benchmarks and metrics, e.g., improving the most recent baseline EditAS by 20.82% and 26.98% in terms of accuracy. AG-RAG also correctly generates 1739 and 2866 unique assertions that all baselines fail to generate, 3.45X and 9.20X more than EditAS.

Paper Structure

This paper contains 26 sections, 10 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Unit Test Generation
  4. DL-based Assertion Generation
  5. Retrieval-based Assertion Generation
  6. Pre-Trained Language Model
  7. Information Retrieval for SE
  8. Approach
  9. Task Formulation
  10. Retrieval-Augmented Assertion Generator
  11. Joint Retriever and Generator Training
  12. Assertion Inference
  13. Usage of AG-RAG
  14. Experimental Setup
  15. Research Questions
  16. ...and 11 more sections

Figures (6)

  • Figure 1: Example of a Test-Assertion Pair (TAP) in $\textit{ATLAS}$
  • Figure 2: The overall framework of AG-RAG
  • Figure 3: The overlaps of the unique generated assertions.
  • Figure 4: Example-1 of assertions generated by approaches
  • Figure 5: Example-2 of assertions generated by approaches
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2