Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance

Elena Bruches, Vadim Alperovich, Dari Baturova, Roman Derunets, Daniil Grebenkin, Georgy Mkrtchyan, Oleg Sedukhin, Mikhail Klementev, Ivan Bondarenko, Nikolay Bushkov, Stanislav Moiseev

TL;DR

TAM-Eval addresses a key gap in automated software testing by evaluating LLMs on end-to-end unit test maintenance across creation, repair, and updating, at the test-file level with repository context. It introduces a 1,539-sample benchmark spanning Python, Java, and Go, with a fully automated, reference-free evaluation pipeline based on pass rate, test coverage, and mutation testing. The framework emphasizes realistic maintenance workflows, including iterative failure feedback, and demonstrates that state-of-the-art LLMs show limited capability on realistic maintenance tasks, though gains are achievable with multiple attempts. By releasing open-source data, code, and infrastructure, TAM-Eval provides a foundation for future advances in automated test maintenance and broader software engineering automation.

Abstract

While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance

TL;DR

TAM-Eval addresses a key gap in automated software testing by evaluating LLMs on end-to-end unit test maintenance across creation, repair, and updating, at the test-file level with repository context. It introduces a 1,539-sample benchmark spanning Python, Java, and Go, with a fully automated, reference-free evaluation pipeline based on pass rate, test coverage, and mutation testing. The framework emphasizes realistic maintenance workflows, including iterative failure feedback, and demonstrates that state-of-the-art LLMs show limited capability on realistic maintenance tasks, though gains are achievable with multiple attempts. By releasing open-source data, code, and infrastructure, TAM-Eval provides a foundation for future advances in automated test maintenance and broader software engineering automation.

Abstract

While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.
Paper Structure (20 sections, 4 equations, 3 figures, 3 tables)

This paper contains 20 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Unit tests generation
  4. Unit tests repairing
  5. Unit tests updating
  6. Unit-testing related benchmarks
  7. Benchmark Construction
  8. Evaluation Framework
  9. Metrics
  10. Evaluation Pipeline
  11. Infrastructure
  12. Experimental Setup
  13. Generation stage
  14. Inference Configuration
  15. Results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Overview of TAM-Eval, a framework and benchmark for evaluating LLMs on unit test maintenance. The pipeline filters GitHub repositories, constructs tasks for test creation, repair, and update with execution-based validation, performs LLM-driven test generation with iterative feedback, and computes Pass Rate, $\Delta$Test Coverage, and $\Delta$Mutation Score.
  • Figure 2: Dynamics across three metrics: Pass Rate, mean $\Delta$ Test Coverage, and mean $\Delta$ Mutation Coverage by attempts
  • Figure 3: Distribution of generated test suites fail reason (attempt@3).