Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving

Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, Haoli Bai

TL;DR

SWE-Lego shows that a lightweight supervised fine-tuning pipeline, when fed with a meticulously constructed hybrid real-and-synthetic dataset and enhanced by step-level error masking and curriculum learning, can attain state-of-the-art results on repository-scale SWE tasks. The method further leverages test-time scaling with a generative verifier to boost performance, achieving 42.2% (8B) and 52.6% (32B) on SWE-bench Verified without Git hacking, and up to 58.8% with TTS@16. The key contributions are the large, executable SWE-Lego dataset (32k task instances, 18k trajectories), the refined SFT procedure that suppresses learning from errors, and the verifier-guided inference strategy that optimizes compute. Together, these results demonstrate a reproducible, open-source, SFT-centric pathway to competitive SWE agents that can rival more complex training paradigms while offering practical advantages in efficiency and robustness.

Abstract

We present SWE-Lego, a supervised fine-tuning (SFT) recipe designed to achieve state-ofthe-art performance in software engineering (SWE) issue resolving. In contrast to prevalent methods that rely on complex training paradigms (e.g., mid-training, SFT, reinforcement learning, and their combinations), we explore how to push the limits of a lightweight SFT-only approach for SWE tasks. SWE-Lego comprises three core building blocks, with key findings summarized as follows: 1) the SWE-Lego dataset, a collection of 32k highquality task instances and 18k validated trajectories, combining real and synthetic data to complement each other in both quality and quantity; 2) a refined SFT procedure with error masking and a difficulty-based curriculum, which demonstrably improves action quality and overall performance. Empirical results show that with these two building bricks alone,the SFT can push SWE-Lego models to state-of-the-art performance among open-source models of comparable size on SWE-bench Verified: SWE-Lego-Qwen3-8B reaches 42.2%, and SWE-Lego-Qwen3-32B attains 52.6%. 3) We further evaluate and improve test-time scaling (TTS) built upon the SFT foundation. Based on a well-trained verifier, SWE-Lego models can be significantly boosted--for example, 42.2% to 49.6% and 52.6% to 58.8% under TTS@16 for the 8B and 32B models, respectively.

SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving

TL;DR

SWE-Lego shows that a lightweight supervised fine-tuning pipeline, when fed with a meticulously constructed hybrid real-and-synthetic dataset and enhanced by step-level error masking and curriculum learning, can attain state-of-the-art results on repository-scale SWE tasks. The method further leverages test-time scaling with a generative verifier to boost performance, achieving 42.2% (8B) and 52.6% (32B) on SWE-bench Verified without Git hacking, and up to 58.8% with TTS@16. The key contributions are the large, executable SWE-Lego dataset (32k task instances, 18k trajectories), the refined SFT procedure that suppresses learning from errors, and the verifier-guided inference strategy that optimizes compute. Together, these results demonstrate a reproducible, open-source, SFT-centric pathway to competitive SWE agents that can rival more complex training paradigms while offering practical advantages in efficiency and robustness.

Abstract

We present SWE-Lego, a supervised fine-tuning (SFT) recipe designed to achieve state-ofthe-art performance in software engineering (SWE) issue resolving. In contrast to prevalent methods that rely on complex training paradigms (e.g., mid-training, SFT, reinforcement learning, and their combinations), we explore how to push the limits of a lightweight SFT-only approach for SWE tasks. SWE-Lego comprises three core building blocks, with key findings summarized as follows: 1) the SWE-Lego dataset, a collection of 32k highquality task instances and 18k validated trajectories, combining real and synthetic data to complement each other in both quality and quantity; 2) a refined SFT procedure with error masking and a difficulty-based curriculum, which demonstrably improves action quality and overall performance. Empirical results show that with these two building bricks alone,the SFT can push SWE-Lego models to state-of-the-art performance among open-source models of comparable size on SWE-bench Verified: SWE-Lego-Qwen3-8B reaches 42.2%, and SWE-Lego-Qwen3-32B attains 52.6%. 3) We further evaluate and improve test-time scaling (TTS) built upon the SFT foundation. Based on a well-trained verifier, SWE-Lego models can be significantly boosted--for example, 42.2% to 49.6% and 52.6% to 58.8% under TTS@16 for the 8B and 32B models, respectively.
Paper Structure (39 sections, 13 figures, 7 tables)

This paper contains 39 sections, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. SWE-Lego Dataset: Combining Real-world and Synthetic Data
  3. Overview
  4. SWE Task Instance Creation and Validation
  5. Repository Collection and Sandbox Construction.
  6. Real-world and Synthetic Task Construction.
  7. Trajectory Rollout and Validation
  8. Validation and Filtering.
  9. Refined Supervised Fine-tuning
  10. Step-level Error Masking
  11. Difficulty-Based Curriculum Learning
  12. Training Details
  13. Error Analysis during Training
  14. Test-time Scaling
  15. Balancing Sequential and Parallel Scaling
  16. ...and 24 more sections

Figures (13)

  • Figure 1: Overview of the performance by SWE-Lego models and its breakdown analysis. (a) SWE-Lego models establish a new frontier on SWE-bench Verified, outperforming same-scale competitors. Notably, our results are based on hack-free evaluation, whereas prior works scores could be inflated by the Git hacking. (b) Our hybrid SWE-Lego dataset delivers the largest boost of +25.6%; subsequent refined SFT adds +3.8%, and TTS contributes +6.2%, together lifting the Qwen3-32B model from 23.2% to 58.8%.
  • Figure 2: Our SWE‑Lego pipeline comprises three stages: environment construction from over 3,000 repositories; hybrid task creation by combining real pull requests with synthetic bugs; and expert‑trajectory generation and curation for SFT.
  • Figure 3: Impact of synthetic data augmentation on data scale and performance. (a) The number of valid expert trajectories increases substantially with the addition of synthetic instances, demonstrating an effective way to expand supervision beyond limited real-world PRs. (b) The resolve rate on SWE-bench Verified rises as more synthetic instances are added per repository, across different repository counts, indicating that hybrid data improves not only dataset scale but also downstream model effectiveness. "Insts./Repo" denotes the average number of instances per repository.
  • Figure 4: Examples of problematic commands or tool interactions: (a) high‑risk commands that can cause Git hacking; (b) view_range parameter mis-specification; (c) ineffective task_tracker.
  • Figure 5: An example of step-level error masking, which maintains the complete trajectory context while selectively masking the loss calculation on incorrect agent responses.
  • ...and 8 more figures