Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Old Meets New: Evaluating the Impact of Regression Tests on SWE Issue Resolution

Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, Martin Hirzel

TL;DR

This work tackles the inefficiency of large regression test suites in debugging open-source SWE issues. It introduces TestPrune, an automated, issue-based test-suite minimization technique that uses LLM-predicted suspicious functions to guide a greedy, coverage-driven selection of a small, highly relevant subset of tests. TestPrune feeds this Subset into reproduction pipelines (e.g., Otter) and patch validation workflows (e.g., Agentless), improving issue reproduction rates by 6.2–9.0% and issue resolution rates by 9.4–12.9% on SWE-Bench Lite and SWE-Bench Verified, while reducing test execution cost to about $0.02–$0.05 per instance. The results demonstrate that reusing a carefully pruned set of regression tests can meaningfully accelerate debugging and repair in agentic SWE pipelines with minimal overhead, representing a practical step toward old tests meeting new issues.

Abstract

Test suites in real-world projects are often large and achieve high code coverage, yet they remain insufficient for detecting all bugs. The abundance of unresolved issues in open-source project trackers highlights this gap. While regression tests are typically designed to ensure past functionality is preserved in the new version, they can also serve a complementary purpose: debugging the current version. Specifically, regression tests can (1) enhance the generation of reproduction tests for newly reported issues, and (2) validate that patches do not regress existing functionality. We present TestPrune, a fully automated technique that leverages issue tracker reports and strategically reuses regression tests for both bug reproduction and patch validation. A key contribution of TestPrune is its ability to automatically minimize the regression suite to a small, highly relevant subset of tests. Due to the predominance of LLM-based debugging techniques, this minimization is essential as large test suites exceed context limits, introduce noise, and inflate inference costs. TestPrune can be plugged into any agentic bug repair pipeline and orthogonally improve overall performance. As a proof of concept, we show that TestPrune leads to a 6.2%-9.0% relative increase in issue reproduction rate within the Otter framework and a 9.4% - 12.9% relative increase in issue resolution rate within the Agentless framework on SWE-Bench Lite and SWE-Bench Verified benchmarks, capturing fixes that were correctly produced by agents but not submitted as final patches. Compared to the benefits, the cost overhead of using TestPrune is minimal, i.e., \$0.02 and \$0.05 per SWE-Bench instance, using GPT-4o and Claude-3.7-Sonnet models, respectively.

When Old Meets New: Evaluating the Impact of Regression Tests on SWE Issue Resolution

TL;DR

This work tackles the inefficiency of large regression test suites in debugging open-source SWE issues. It introduces TestPrune, an automated, issue-based test-suite minimization technique that uses LLM-predicted suspicious functions to guide a greedy, coverage-driven selection of a small, highly relevant subset of tests. TestPrune feeds this Subset into reproduction pipelines (e.g., Otter) and patch validation workflows (e.g., Agentless), improving issue reproduction rates by 6.2–9.0% and issue resolution rates by 9.4–12.9% on SWE-Bench Lite and SWE-Bench Verified, while reducing test execution cost to about 0.05 per instance. The results demonstrate that reusing a carefully pruned set of regression tests can meaningfully accelerate debugging and repair in agentic SWE pipelines with minimal overhead, representing a practical step toward old tests meeting new issues.

Abstract

Test suites in real-world projects are often large and achieve high code coverage, yet they remain insufficient for detecting all bugs. The abundance of unresolved issues in open-source project trackers highlights this gap. While regression tests are typically designed to ensure past functionality is preserved in the new version, they can also serve a complementary purpose: debugging the current version. Specifically, regression tests can (1) enhance the generation of reproduction tests for newly reported issues, and (2) validate that patches do not regress existing functionality. We present TestPrune, a fully automated technique that leverages issue tracker reports and strategically reuses regression tests for both bug reproduction and patch validation. A key contribution of TestPrune is its ability to automatically minimize the regression suite to a small, highly relevant subset of tests. Due to the predominance of LLM-based debugging techniques, this minimization is essential as large test suites exceed context limits, introduce noise, and inflate inference costs. TestPrune can be plugged into any agentic bug repair pipeline and orthogonally improve overall performance. As a proof of concept, we show that TestPrune leads to a 6.2%-9.0% relative increase in issue reproduction rate within the Otter framework and a 9.4% - 12.9% relative increase in issue resolution rate within the Agentless framework on SWE-Bench Lite and SWE-Bench Verified benchmarks, capturing fixes that were correctly produced by agents but not submitted as final patches. Compared to the benefits, the cost overhead of using TestPrune is minimal, i.e., \0.05 per SWE-Bench instance, using GPT-4o and Claude-3.7-Sonnet models, respectively.
Paper Structure (26 sections, 2 equations, 10 figures, 3 tables, 2 algorithms)

This paper contains 26 sections, 2 equations, 10 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Statement and Challenges
  3. Approach
  4. TestPrune
  5. Suspicious Function Localization.
  6. Test File Retrieval and Coverage Generation.
  7. Greedy Algorithm.
  8. Reproduction Test
  9. Patch Selection and Validation
  10. Evaluation
  11. Research Questions
  12. Experiment Setup.
  13. Patch Selection and Validation Workflow: Agentless
  14. Reproduction Test Generation Workflow: Otter
  15. Evaluation Benchmarks: SWE-Bench Verified and SWE-Bench Lite
  16. ...and 11 more sections

Figures (10)

  • Figure 1: Overview of TestPrune and clients, reproduction test generation (yellow) and patch generation/selection (green), in the end-to-end pipeline of fixing open-source issues
  • Figure 2: Suspicious function localization
  • Figure 3: Prompt template of test file selection
  • Figure 4: Example of greedy-additional algorithm (a) and greedy-total algorithm (b). $M$ denote suspicious functions, $S$ are their lines, and $T$ denotes tests
  • Figure 5: Comparison between the original test suite and TestPrune-minimized regression tests in terms of (a) test size and (b) test runtime. The y-axis scales differ across the configurations for varying magnitudes.
  • ...and 5 more figures