Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, Martin Hirzel
TL;DR
This work tackles generating reproduction tests from SWE issues when resolving code patches are unavailable. It introduces e-Otter++, which augments Otter++ with execution feedback, issue description morphs, and test-generation context masks, and leverages surrogate code patches to guide test selection. Empirical results on SWT-bench Lite, TDD-Bench Verified, and SWE-rebench show substantial improvements, achieving a new state-of-the-art $F\!\to\!\!P$ rate (63.0% on TDD-Bench Verified) and demonstrating that heterogeneous prompting and execution-informed repair are key drivers. The approach also enhances SWE-patch validation and supports potential integration with real-world SWE agents, advancing TDD-driven quality assurance in software engineering.
Abstract
A software engineering issue (SWE issue) is easier to resolve when accompanied by a reproduction test. Unfortunately, most issues do not come with functioning reproduction tests, so this paper explores how to generate them automatically. The primary challenge in this setting is that the code to be tested is either missing or wrong, as evidenced by the existence of the issue in the first place. This has held back test generation for this setting: without the correct code to execute, it is difficult to leverage execution feedback to generate good tests. This paper introduces novel techniques for leveraging execution feedback to get around this problem, implemented in a new reproduction test generator called e-Otter++. Experiments show that e-Otter++ represents a leap ahead in the state-of-the-art for this problem, generating tests with an average fail-to-pass rate of 63% on the TDD-Bench Verified benchmark.