SweRank: Software Issue Localization with Code Ranking

TL;DR

SweRank reframes software issue localization as a retrieve-and-rerank problem to address the high latency and cost of agent-based LLM systems. It introduces SweRankEmbed as a bi-encoder retriever and SweRankLLM as a listwise LLM reranker, trained on the SweLoc dataset built from real-world GitHub PRs with consistency filtering and hard-negative mining. Empirical results on SWE-Bench-Lite and LocBench show state-of-the-art localization accuracy, with substantial cost advantages over closed-source agent systems. The SweLoc dataset also proves effective for finetuning a variety of retrievers and rerankers, underscoring its value for advancing open, scalable approaches to issue localization.

Abstract

Software issue localization, the task of identifying the precise code locations (files, classes, or functions) relevant to a natural language issue description (e.g., bug report, feature request), is a critical yet time-consuming aspect of software development. While recent LLM-based agentic approaches demonstrate promise, they often incur significant latency and cost due to complex multi-step reasoning and relying on closed-source LLMs. Alternatively, traditional code ranking models, typically optimized for query-to-code or code-to-code retrieval, struggle with the verbose and failure-descriptive nature of issue localization queries. To bridge this gap, we introduce SweRank, an efficient and effective retrieve-and-rerank framework for software issue localization. To facilitate training, we construct SweLoc, a large-scale dataset curated from public GitHub repositories, featuring real-world issue descriptions paired with corresponding code modifications. Empirical results on SWE-Bench-Lite and LocBench show that SweRank achieves state-of-the-art performance, outperforming both prior ranking models and costly agent-based systems using closed-source LLMs like Claude-3.5. Further, we demonstrate SweLoc's utility in enhancing various existing retriever and reranker models for issue localization, establishing the dataset as a valuable resource for the community.

Figures (7)

• Figure 1: Comparison of localization performance versus cost per instance on SWE-Bench-Lite. Our proposed SweRankEmbed retriever and SweRankLLM reranker models achieve superior accuracy at a significantly lower cost compared to contemporary agent-based methods for issue localization.
• Figure 2: Overview of SweLoc data construction pipeline, illustrating the three main stages.
• Figure 3: (Left) Distribution of query lengths in the SweLoc dataset. The red dashed line indicates a mean query length of 382.56 tokens, underscoring the detailed nature typical of software issue reports. (Right) Distribution of the number of (a) files, (b) modules, and (c) functions modified per GitHub issue. This highlights that while many localizations are concentrated, a significant number span multiple code units, particularly at finer granularities.
• Figure 4: Function localization performance comparison across different categories within LocBench. Our approach, which incorporates the SweRankLLM reranker, considerably outperforms other Agent-based methods powered by Claude-3.5.
• Figure 5: Effect of SweLoc data filtering on SWE-Bench-Lite performance. While accuracy does improve from discarding training examples where the positive instance's rank among negative candidates $>$$K$, no filtering (i.e. $K$=None) considerably hurts performance.