SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents

TL;DR

SWE-PolyBench presents a repository-level, multi-language, execution-based benchmark for evaluating coding agents across Java, JavaScript, TypeScript, and Python. It expands beyond SWE-Bench by introducing a larger, more diverse task set (2110 instances across 21 repos) and a stratified subset SWE-PolyBench500, plus novel CST-based retrieval metrics to assess code navigation. The study reveals language- and task-dependent performance with Python-leading results and notable difficulties for complex, multi-file edits, underscoring gaps in current open-source agents. By providing datasets, a runtime harness, and CST-focused evaluation, SWE-PolyBench aims to drive the development of more versatile and robust AI coding assistants for real-world software engineering.

Abstract

Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench

Figures (6)

• Figure 1: Pass rates of coding agents across programming languages (left) and across subsets of different complexities based on syntax tree nodes. The right plot categorizes changes by type (class or function) and scope (single or multiple), with "No nodes" indicating no class or function changes and "Mixed" requiring both.
• Figure 2: Overview of our dataset generation pipeline. We start by collecting pull requests (PRs) that close an issue from popular repositories across our four target languages. After applying a metadata filter, we then create containerized environments for test execution. Test outcomes are compared before and after patch application. A test is considered a fail-to-pass (F2P) if it initially fails but passes after applying the patch, a pass-to-pass (P2P) test is defined accordingly. The final test-based filter selects PRs that have at least one F2P test and are considered "testable" following the definition in \ref{['sec:runtime setup']}.
• Figure 3: Classification of SWE-PolyBench issue descriptions with respect to their descriptiveness (left plot), hints at the solution (middle plot), and information on the localization of the issue (right plot).
• Figure 4: Classification of SWE-PolyBench500 issue descriptions with respect to their descriptiveness (left plot), hints a the solution (middle plot) and information on the localization of the issue (right plot
• Figure 5: Pass rates of instances with respect to their descriptiveness (left plot), hints a the solution (middle plot) and information on the localization of the issue (right plot).