Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PROZE: Generating Parameterized Unit Tests Informed by Runtime Data

Deepika Tiwari, Yogya Gamage, Martin Monperrus, Benoit Baudry

TL;DR

PROZE tackles the challenge of generating useful parameterized unit tests by transforming conventional unit tests into PUTs using runtime data. It collects real inputs from test and production executions, isolates single assertions from CUTs, and produces PUTs with argument providers drawn from observed inputs. The approach yields a taxonomy of PUTs (strongly-coupled, decoupled, falsifiably-coupled) and demonstrates substantial increases in input-space coverage across five real-world Java modules, with 217 PUTs showing broader valid input ranges. The work provides a practical, automatically-usable method with a public Java implementation, offering a path toward more expressive and scalable testing in real projects.

Abstract

Typically, a conventional unit test (CUT) verifies the expected behavior of the unit under test through one specific input / output pair. In contrast, a parameterized unit test (PUT) receives a set of inputs as arguments, and contains assertions that are expected to hold true for all these inputs. PUTs increase test quality, as they assess correctness on a broad scope of inputs and behaviors. However, defining assertions over a set of inputs is a hard task for developers, which limits the adoption of PUTs in practice. In this paper, we address the problem of finding oracles for PUTs that hold over multiple inputs. We design a system called PROZE, that generates PUTs by identifying developer-written assertions that are valid for more than one test input. We implement our approach as a two-step methodology: first, at runtime, we collect inputs for a target method that is invoked within a CUT; next, we isolate the valid assertions of the CUT to be used within a PUT. We evaluate our approach against 5 real-world Java modules, and collect valid inputs for 128 target methods from test and field executions. We generate 2,287 PUTs, which invoke the target methods with a significantly larger number of test inputs than the original CUTs. We execute the PUTs and find 217 that provably demonstrate that their oracles hold for a larger range of inputs than envisioned by the developers. From a testing theory perspective, our results show that developers express assertions within CUTs that are general enough to hold beyond one particular input.

PROZE: Generating Parameterized Unit Tests Informed by Runtime Data

TL;DR

PROZE tackles the challenge of generating useful parameterized unit tests by transforming conventional unit tests into PUTs using runtime data. It collects real inputs from test and production executions, isolates single assertions from CUTs, and produces PUTs with argument providers drawn from observed inputs. The approach yields a taxonomy of PUTs (strongly-coupled, decoupled, falsifiably-coupled) and demonstrates substantial increases in input-space coverage across five real-world Java modules, with 217 PUTs showing broader valid input ranges. The work provides a practical, automatically-usable method with a public Java implementation, offering a path toward more expressive and scalable testing in real projects.

Abstract

Typically, a conventional unit test (CUT) verifies the expected behavior of the unit under test through one specific input / output pair. In contrast, a parameterized unit test (PUT) receives a set of inputs as arguments, and contains assertions that are expected to hold true for all these inputs. PUTs increase test quality, as they assess correctness on a broad scope of inputs and behaviors. However, defining assertions over a set of inputs is a hard task for developers, which limits the adoption of PUTs in practice. In this paper, we address the problem of finding oracles for PUTs that hold over multiple inputs. We design a system called PROZE, that generates PUTs by identifying developer-written assertions that are valid for more than one test input. We implement our approach as a two-step methodology: first, at runtime, we collect inputs for a target method that is invoked within a CUT; next, we isolate the valid assertions of the CUT to be used within a PUT. We evaluate our approach against 5 real-world Java modules, and collect valid inputs for 128 target methods from test and field executions. We generate 2,287 PUTs, which invoke the target methods with a significantly larger number of test inputs than the original CUTs. We execute the PUTs and find 217 that provably demonstrate that their oracles hold for a larger range of inputs than envisioned by the developers. From a testing theory perspective, our results show that developers express assertions within CUTs that are general enough to hold beyond one particular input.
Paper Structure (25 sections, 1 figure, 4 tables)

This paper contains 25 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Conventional Unit Tests (CUT)
  4. Parameterized Unit Tests (PUT)
  5. proze
  6. Overview
  7. Capturing PUT inputs
  8. Automatic generation of PUTs
  9. Parameterize over a target method
  10. Isolate each assertion
  11. Generate argument provider method
  12. Classes of generated PUTs
  13. Illustrative example
  14. Implementation details
  15. Experimental Methodology
  16. ...and 10 more sections

Figures (1)

  • Figure 1: An overview of proze, a novel technique for the automated transformation of a developer-written conventional unit test (CUT) into parameterized unit tests (PUTs). Given a project and its original test suite composed of a set of CUTs, proze generates a set of PUTs, which are classified as either strongly-coupled, decoupled or falsifiably-coupled, depending on how much the oracle is tied to the original input. The CUT on the top right is transformed into corresponding PUTs on the bottom right. Each PUT is fed with arguments collected at runtime with bespoke monitoring.