Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Test Case Repair Using Language Models

Ahmadreza Saboor Yaraghi, Darren Holden, Nafiseh Kahani, Lionel Briand

TL;DR

TaRGET reframes test case repair as a code translation task, leveraging pre-trained code language models (CLMs) and a carefully engineered repair context to automatically repair broken tests. It introduces TaRBench, a large-scale benchmark of 45,373 broken-test repairs across 59 open-source Java projects, and demonstrates that fine-tuning CLMs with repair-context data achieves EM $66.1\%$ and PR $80\%$, outperforming baselines by substantial margins. The approach hinges on identifying repair context (method/class hunks, call graphs) and prioritizing relevant information, then generating repairs via encoder-decoder or decoder-only models using four IO formats. The study also provides a trustworthiness predictor, analyzes repair characteristics, explores data-size effects, and shows that TaRGET generalizes across unseen projects, offering a practical, scalable solution for test maintenance in real-world software development.

Abstract

Ensuring the quality of software systems through testing is essential, yet maintaining test cases poses significant challenges and costs. The need for frequent updates to align with the evolving system under test often entails high complexity and cost for maintaining these test cases. Further, unrepaired broken test cases can degrade test suite quality and disrupt the software development process, wasting developers' time. To address this challenge, we present TaRGET (Test Repair GEneraTor), a novel approach leveraging pre-trained code language models for automated test case repair. TaRGET treats test repair as a language translation task, employing a two-step process to fine-tune a language model based on essential context data characterizing the test breakage. To evaluate our approach, we introduce TaRBench, a comprehensive benchmark we developed covering 45,373 broken test repairs across 59 open-source projects. Our results demonstrate TaRGET's effectiveness, achieving a 66.1% exact match accuracy. Furthermore, our study examines the effectiveness of TaRGET across different test repair scenarios. We provide a practical guide to predict situations where the generated test repairs might be less reliable. We also explore whether project-specific data is always necessary for fine-tuning and if our approach can be effective on new projects.

Automated Test Case Repair Using Language Models

TL;DR

TaRGET reframes test case repair as a code translation task, leveraging pre-trained code language models (CLMs) and a carefully engineered repair context to automatically repair broken tests. It introduces TaRBench, a large-scale benchmark of 45,373 broken-test repairs across 59 open-source Java projects, and demonstrates that fine-tuning CLMs with repair-context data achieves EM and PR , outperforming baselines by substantial margins. The approach hinges on identifying repair context (method/class hunks, call graphs) and prioritizing relevant information, then generating repairs via encoder-decoder or decoder-only models using four IO formats. The study also provides a trustworthiness predictor, analyzes repair characteristics, explores data-size effects, and shows that TaRGET generalizes across unseen projects, offering a practical, scalable solution for test maintenance in real-world software development.

Abstract

Ensuring the quality of software systems through testing is essential, yet maintaining test cases poses significant challenges and costs. The need for frequent updates to align with the evolving system under test often entails high complexity and cost for maintaining these test cases. Further, unrepaired broken test cases can degrade test suite quality and disrupt the software development process, wasting developers' time. To address this challenge, we present TaRGET (Test Repair GEneraTor), a novel approach leveraging pre-trained code language models for automated test case repair. TaRGET treats test repair as a language translation task, employing a two-step process to fine-tune a language model based on essential context data characterizing the test breakage. To evaluate our approach, we introduce TaRBench, a comprehensive benchmark we developed covering 45,373 broken test repairs across 59 open-source projects. Our results demonstrate TaRGET's effectiveness, achieving a 66.1% exact match accuracy. Furthermore, our study examines the effectiveness of TaRGET across different test repair scenarios. We provide a practical guide to predict situations where the generated test repairs might be less reliable. We also explore whether project-specific data is always necessary for fine-tuning and if our approach can be effective on new projects.
Paper Structure (62 sections, 8 equations, 23 figures, 11 tables)

This paper contains 62 sections, 8 equations, 23 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Language Models
  4. Language Model Architecture
  5. Pre-Training, Fine-Tuning, and Inference of Code Language Models
  6. Approach: TaRGET
  7. Repair Context Identification and Prioritization
  8. Repair Context Identification
  9. Repair Context Prioritization
  10. Input Creation and Repair Generation
  11. The Input
  12. The Output
  13. Input-Output Format Selection
  14. Repair Generation
  15. Study Design and Results
  16. ...and 47 more sections

Figures (23)

  • Figure 1: Two real-world broken test examples causing compilation and runtime errors.
  • Figure 2: An overview of TaRGET given a broken test case and two versions ($V_i$ and $V_{i+1}$) of the system under test (SUT).
  • Figure 3: Test case repair example
  • Figure 4: Example input format with word-level hunk representation and the expected code sequence output.
  • Figure 5: Example of the line-level representation of a code hunk ($h_2$ in our running example) in the input.
  • ...and 18 more figures