SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories

TL;DR

SWE-Bench++ tackles the scalability and realism gaps in repository-level benchmarks by automating the generation of executable tasks from GitHub PRs across 11 languages. It introduces constrained environment and adaptive log-parsing synthesis, a state-differential test oracle to distinguish bug fixes from feature requests, and a four-layer AutoQA pipeline with optional human verification. A hint-guided trajectory synthesis mechanism converts model-breaking instances into high-signal training data, enabling measurable cross-lingual improvements upon fine-tuning. The framework yields a large, contamination-aware, multilingual benchmark and demonstrates tangible benefits in model evaluation and training, highlighting its potential to drive generalization across languages and build systems. Overall, SWE-Bench++ provides a scalable, living benchmark that aligns evaluation with real-world software evolution while supporting targeted improvements via training trajectories.

Abstract

Benchmarks like SWE-bench have standardized the evaluation of Large Language Models (LLMs) on repository-level software engineering tasks. However, these efforts remain limited by manual curation, static datasets, and a focus on Python-based bug fixes. We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects. Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11 languages. SWE-Bench++ turns GitHub pull requests (PRs) into reproducible, execution-based tasks via four stages: programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. A final hint-guided trajectory synthesis step converts instances that strong models fail on into training trajectories. Our initial benchmark consists of 11,133 instances from 3,971 repositories across 11 languages. On a subset of 1,782 instances of this benchmark, today's strongest models perform as follows: claude-sonnet-4.5 achieves 36.20% pass@10, gpt-5-2025-08-07 34.57%, gemini/gemini-2.5-pro 24.92%, and gpt-4o 16.89%. We further demonstrate the utility of our dataset by showing that fine-tuning on SWE-Bench++ instances yields measurable improvements on the SWE-bench Multilingual benchmark. SWE-Bench++ provides a scalable, multilingual benchmark for evaluating and improving repository-level code generation.

Figures (2)

• Figure 1: The SWE-Bench++ Framework. Unlike static benchmarks, our pipeline uses constrained neural synthesis to generate reproducible Docker environments and adaptive log parsers across 11 languages. A three‑state, state‑differential oracle automatically classifies tasks as bug fixes or feature requests, producing verified benchmark instances and hint‑guided training trajectories at scale.
• Figure 2: An overall view of the dataset.