Repository Structure-Aware Training Makes SLMs Better Issue Resolver

TL;DR

This work addresses the gap in repository-level problem solving by small language models (SLMs) through Repository Structure-Aware Training (ReSAT). By constructing two data types—localization data across file, function, and line granularity, and code-edit data—from a large set of open-source GitHub issues and PRs, ReSAT fine-tunes SLMs to better understand repository structure and perform context-aware edits. Evaluations on SWE-Bench-verified and RepoQA demonstrate that ReSAT improves issue-resolving performance and long-context code understanding, with ablations confirming the complementary benefits of localization and code-edit data. The approach offers a practical path to bring competitive repository-level capabilities to open-source SLMs while reducing reliance on costly LLMs.

Abstract

Language models have been applied to various software development tasks, but the performance varies according to the scale of the models. Large Language Models (LLMs) outperform Small Language Models (SLMs) in complex tasks like repository-level issue resolving, but raise concerns about privacy and cost. In contrast, SLMs are more accessible but under-perform in complex tasks. In this paper, we introduce ReSAT (Repository Structure-Aware Training), construct training data based on a large number of issues and corresponding pull requests from open-source communities to enhance the model's understanding of repository structure and issue resolving ability. We construct two types of training data: (1) localization training data, a multi-level progressive localization data to improve code understanding and localization capability; (2) code edit training data, which improves context-based code editing capability. The evaluation results on SWE-Bench-verified and RepoQA demonstrate that ReSAT effectively enhances SLMs' issue-resolving and repository-level long-context understanding capabilities.

Figures (5)

• Figure 1: Typical issue resolving frameworks, both of them first localize the code snippets related to the issue, then edit the snippets to resolve the issue.
• Figure 2: ReSAT training data scraping, construction and training pipeline. We first scrape issues and PRs from open-source repositories, then construct ReSAT localization and code edit data, and apply ReSAT training on SLMs.
• Figure 3: Ablation study on training datasets. The first four figures show the metrics on SWE-bench-verified. The last one shows the average accuracy on RepoQA.
• Figure 4: Case study.
• Figure 5: Comparison results between RAG-SWE with ReSAT-trained SLMs and LLMs.