BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills

TL;DR

BugPilot introduces FeatAdd and BugInstruct to generate synthetic bugs via agentic feature development, producing realistic and diverse failures that better train SWE agents. FeatAdd-based bugs yield more data-efficient learning and, when combined with supervised fine-tuning and reinforcement learning, achieve state-of-the-art SWE-Bench Verified results for 32B and 14B models. The approach demonstrates strong improvements in data efficiency and generalization over prior synthetic bug datasets, underscoring the value of realistic bug-generation workflows. The work suggests scalable, open-weight SWE agents can be advanced by leveraging agent-generated training data and interfaces for broader domain adaptation.

Abstract

High quality bugs are key to training the next generation of language model based software engineering (SWE) agents. We introduce a novel method for synthetic generation of difficult and diverse bugs. Our method instructs SWE Agents to introduce a feature into the codebase whereby they may unintentionally break tests, resulting in bugs. Prior approaches often induce an out-of-distribution effect by generating bugs intentionally (e.g. by introducing local perturbation to existing code), which does not reflect realistic development processes. We perform qualitative analysis to demonstrate that our approach for generating bugs more closely reflects the patterns found in human-authored edits. Through extensive experiments, we demonstrate that our bugs provide more efficient training data for supervised fine-tuning, outperforming other bug datasets by 2% with half the training data (1.2k vs. 3k bugs). We train on our newly generated bugs in addition to existing bug datasets to get FrogBoss a state-of-the-art 32B parameter model on SWE-bench Verified with a pass@1 of 54.6% and FrogMini a state-of-the-art 14B model on SWE-bench Verified with a pass@1 of 45.3% on SWE-bench Verified averaged over three seeds.

Figures (4)

• Figure 1: Comparison to previous SoTA results. We train FrogBoss with our collected data including our new FeatAdd datasets, and FrogBoss achieves 54.6% pass@1 averaged over three seeds. With pass@3 we achieve a score of 67.4%, outperforming Claude Sonnet 4, illustrating the performance gains we could get with a good verifier. The minimal test time scaling refers to the strategy of selecting the shortest trajectory among three rollouts. Pass@short refers to taking the shortest rollout from a group of three rollouts on a given problem.
• Figure 2: Illustration of our BugPilot pipeline. First, we instruct SWE Agent NEURIPS2024_5a7c9475 with Claude Sonnet 4 to introduce bugs, either through deliberate attempts or by attempting to add a feature. Then, we check whether these modifications resulted in the tests for the repository failing. If the tests fail, then we add this to our dataset of bugs. Otherwise, we ask the model to continue changing the code until the tests fail.
• Figure 3: Contrasting approaches to adding bugs. On the left, our FeatAdd approach first attempts to implement a token counting feature in the repository. This results in large changes across multiple files as well as a test case failure arising from a seemingly unrelated part of the repository. In contrast on the right, approaches like SWE-Smith and our proposed baseline BugInstruct make local perturbations to the code which cause related tests to fail. We can see that FeatAdd more closely resembles how bugs arise during the development process, where test failures can occur due to complex interactions between changes.
• Figure 4: Distribution of common bug types across different bug datasets.FeatAdd demonstrates the most even distribution of bugs compared to prior work as well as our agentic baseline BugInstruct. SWE-Smith bugs shows a particular skew towards logic and conditional bugs. This can be explained by the rule based and local nature of the SWE-Smith generation process. Most bugs generated by BugInstruct are state consistency / bookkeeping / caching bugs or copy semantics / mutability aliasing / in-place mutation of inputs bugs. The distribution of FeatAdd bugs is similar to R2E-Gym and SWE-Bench, which is closest to the human authored edit distribution found in real repositories.