Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, Wenhu Chen

Abstract

Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a ~13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow.

SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

Abstract

Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a ~13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow.
Paper Structure (38 sections, 4 equations, 10 figures, 11 tables, 1 algorithm)

This paper contains 38 sections, 4 equations, 10 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. SWE-QA-Pro Bench
  3. Data Sourcing and Taxonomy
  4. Data Synthesis and Sampling
  5. Data Filtering and Difficulty Calibration
  6. Data Validation and Statistics
  7. SWE-QA-Pro Agent
  8. Agent Workflow
  9. Agentic Training Recipe
  10. Experiments
  11. Experimental Setup
  12. Main Results
  13. Tool Usage Analysis
  14. Training Strategy Analysis
  15. Related Work
  16. ...and 23 more sections

Figures (10)

  • Figure 1: SWE-QA-Pro Benchmark and Training Pipeline.
  • Figure 2: t-SNE visualization of semantic distributions. (a) Comparison of the original issue spaces, showing the broad coverage of SWE-Rebench compared to the manually curated SWE-Verified. (b) The distribution of our synthesized datasets (Training and Test) demonstrates high diversity and alignment with the semantic clusters of existing benchmarks.
  • Figure 3: Difficulty Comparison between SWE-QA and SWE-QA-Pro. Higher difficulty indicates harder questions.
  • Figure 4: Tool usage behavior across models on SWE-QA-Pro.
  • Figure 5: Effect of Training Strategy. We compare SFT at different scales and a two-stage SFT→RLAIF setting.
  • ...and 5 more figures