SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

TL;DR

SWT-Bench addresses the need for real-world, issue-reproducing tests by offering a Python-focused benchmark derived from SWE-Bench, with real issues, ground-truth patches, and golden tests. It demonstrates that Code Agents, when adapted for test generation, can outperform non-agent baselines in producing reproducing tests and can effectively filter code fixes via test outcomes, doubling precision in some setups. The work provides a robust evaluation framework with metrics for reproduction success and patch coverage, enabling nuanced analysis beyond simple accuracy and revealing complementarity among diverse approaches. Overall, SWT-Bench advances the study of test generation and its role in improving automatic code repair, with practical implications for improving software quality and developer productivity.

Abstract

Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench

Figures (16)

• Figure 1: Evaluation of an SWT-Bench instance. Given an issue description in natural language and the corresponding codebase, the task is to generate tests that reproduce the issue. We considered a test to reproduce the issue if it fails on the codebase before the pull request (PR) is accepted, i.e., before the golden patch is applied, but passes after. We call this a fail-to-pass test ($F \! \to \! P$).
• Figure 2: Distribution of SWT-Bench instances over GitHub repositories.
• Figure 3: Illustration of change coverage $\Delta\mathcal{C}$ of the generated tests $T$, given the test suite $T^R$ of the original code base $R$, the golden patch $X^*$, and the golden tests $T^*$.
• Figure 4: Comparison of the default unified diff format (left) and our fault-tolerant version (right).
• Figure 5: Distribution of success rate ($\mathcal{S}$) across issue description lengths in # tokens