Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Deep Semantics for Test Completion

Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, Milos Gligoric

TL;DR

The paper introduces test completion and TeCo, a transformer model that uses six static code-semantics signals plus execution-based reranking to automatically predict the next statement in a Java test method. It builds a large corpus of 130,934 test methods from 1,270 open-source projects and fine-tunes CodeT5 for the task, using beam search and a reranking stage that prefers compilable and runnable outputs. Across intrinsic lexical metrics and a functional correctness measure (compile/run), TeCo consistently outperforms syntax-only baselines and state-of-the-art test-oracle models, and shows clear benefits from execution-aware reranking and diverse code-semantics signals. The work demonstrates that incorporating execution semantics into ML models significantly improves automated test writing and provides a public corpus to advance research in ML for software testing.

Abstract

Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is hard to do with only syntax-level data that existing code completion models use. TeCo extracts and uses six kinds of code semantics data, including the execution result of prior statements and the execution context of the test method. To provide a testbed for this new task, as well as to evaluate TeCo, we collect a corpus of 130,934 test methods from 1,270 open-source Java projects. Our results show that TeCo achieves an exact-match accuracy of 18, which is 29% higher than the best baseline using syntax-level data only. When measuring functional correctness of generated next statement, TeCo can generate runnable code in 29% of the cases compared to 18% obtained by the best baseline. Moreover, TeCo is significantly better than prior work on test oracle generation.

Learning Deep Semantics for Test Completion

TL;DR

The paper introduces test completion and TeCo, a transformer model that uses six static code-semantics signals plus execution-based reranking to automatically predict the next statement in a Java test method. It builds a large corpus of 130,934 test methods from 1,270 open-source projects and fine-tunes CodeT5 for the task, using beam search and a reranking stage that prefers compilable and runnable outputs. Across intrinsic lexical metrics and a functional correctness measure (compile/run), TeCo consistently outperforms syntax-only baselines and state-of-the-art test-oracle models, and shows clear benefits from execution-aware reranking and diverse code-semantics signals. The work demonstrates that incorporating execution semantics into ML models significantly improves automated test writing and provides a public corpus to advance research in ML for software testing.

Abstract

Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is hard to do with only syntax-level data that existing code completion models use. TeCo extracts and uses six kinds of code semantics data, including the execution result of prior statements and the execution context of the test method. To provide a testbed for this new task, as well as to evaluate TeCo, we collect a corpus of 130,934 test methods from 1,270 open-source Java projects. Our results show that TeCo achieves an exact-match accuracy of 18, which is 29% higher than the best baseline using syntax-level data only. When measuring functional correctness of generated next statement, TeCo can generate runnable code in 29% of the cases compared to 18% obtained by the best baseline. Moreover, TeCo is significantly better than prior work on test oracle generation.
Paper Structure (24 sections, 4 equations, 6 figures, 19 tables)

This paper contains 24 sections, 4 equations, 6 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Task
  3. Extraction of Code Semantics
  4. TeCo's Deep Learning Model
  5. Encoder-Decoder Transformer Model
  6. Fine-tuning
  7. Evaluation
  8. Reranking by Execution
  9. Corpus
  10. Experiments Setup
  11. TeCo Models
  12. Baseline Models
  13. Subsets of the Evaluation Set
  14. Evaluation Metrics
  15. Results
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Example of test completion: given the code under test (represented by the method under test), test method signature, and prior statements, the goal is to generate the next statement. Code from $\mathtt{sharneng/gm4java}$ in class $\mathtt{GMOperationTest}$.
  • Figure 2: TeCo's workflow of using static analysis to extract code semantics.
  • Figure 3: Some kinds of code semantics ( and , highlighted in the first two blue boxes) extracted for the example in \ref{['fig:example-data']}, which help TeCo generate the correct next statement (highlighted in the last green box).
  • Figure 4: TeCo's model architecture.
  • Figure 5: Procedure of detecting whether a generated statement is compilable and runnable.
  • ...and 1 more figures