Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories

Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava, Valerie Chen, Debojyoti Dutta

TL;DR

This work proposes SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests from natural language issue descriptions and shows consistent improvements with increased inference-time compute, more data, and larger models.

Abstract

Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development -- "test first, write code later", and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10\% in success rate and 21\% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain.

SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories

TL;DR

This work proposes SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests from natural language issue descriptions and shows consistent improvements with increased inference-time compute, more data, and larger models.

Abstract

Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development -- "test first, write code later", and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10\% in success rate and 21\% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain.
Paper Structure (29 sections, 3 figures, 8 tables)

This paper contains 29 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Issue Reproduction Task
  5. Workflow for Issue Reproduction
  6. Code Localization
  7. Code Editing
  8. Dataset Curation and Training Methodology
  9. Inference Scaffold
  10. Experimental Setup
  11. Results
  12. Retrieval Performance for Code Localization
  13. Analysis
  14. Effect of scaling inference-time compute on model performance
  15. Comparing Edit-only Performance of LLMs with Oracle Retrieval
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Our proposed SWE-Tester pipeline. Given an issue description and the pre-PR repository (which has this issue/bug), we first retrieve defective source code and relevant test file (code localization). Next, we edit the test file to augment it with reproduction tests (code editing).
  • Figure 2: Increasing inference-time compute (i.e. no. of sampled patches) improves both success rate and change coverage of the fine-tuned Qwen2.5-Coder Instruct 7B, 14B, and 32B LLMs on SWT-Bench Verified.
  • Figure 3: Scaling effects of increasing the amount data used when training Qwen2.5-Coder Instruct 14B and 32B LLMs.