PerfBench: Can Agents Resolve Real-World Performance Bugs?

TL;DR

PerfBench presents the first repo-scale benchmark for evaluating software engineering agents on real-world performance bug fixes in .NET, enabling agents to generate and validate their own performance benchmarks. The study shows baseline agents struggle with perf optimization, achieving around 3% success, while a performance-aware variant reaches about 20%, highlighting the value of explicit benchmarking workflows and tooling. By combining a curated 81-task dataset with an automated benchmarking harness, PerfBench offers a demanding testbed that emphasizes measurable efficiency gains and regression-free fixes, driving progress toward smarter, performance-conscious agents. The work also outlines limitations and avenues for future improvements across language scope, benchmarking reliability, and broader performance metrics.

Abstract

Performance bugs are inefficiencies in software that waste computational resources without causing functional failures, making them particularly challenging to detect and fix. While recent advances in Software Engineering agents have shown promise in automated bug fixing, existing benchmarks primarily focus on functional correctness and fail to evaluate agents' abilities to identify and resolve non-functional issues like performance bugs. We introduce PerfBench, a benchmark comprising 81 real-world performance bug-fixing tasks from popular .NET repositories on GitHub. Unlike existing benchmarks that rely on pre-existing test suites, PerfBench features a novel evaluation harness that allows agents to generate their own performance benchmarks and validates fixes by comparing execution metrics collected for developer fix and agent fix. Each task in PerfBench is derived from actual developer fixes linked to performance-related issues, which are then verified by human experts, ensuring real-world relevance. Our evaluation reveals that current state-of-the-art coding agents struggle with performance optimization tasks, with baseline OpenHands agent achieving only a ~3% success rate on our benchmark. We develop OpenHands-Perf-Agent, which incorporates performance-aware tooling and instructions and achieves a ~20% success rate on the benchmark. We show that by ensuring the agent has proper instructions to benchmark its changes and tooling for benchmark output processing, we can improve the agent performance significantly, but room for improvement still remains. PerfBench provides a challenging test set for furthering the capabilities of agents in fixing performance issues.

Figures (5)

• Figure 1: Distribution of C# repositories in the benchmark.
• Figure 2: Example task from PerfBench showing the metadata collected in our benchmark. The task given to the agent includes repository context and the original GitHub issue describing an OutOfMemoryException.
• Figure 3: A sample BenchmarkDotNet output table showing execution time statistics, memory allocation and Garbage Collection (GC) metrics across different tests within the test suite. Our evaluation harness uses this to determine whether the code changes improve performance.
• Figure 4: Prompt template differences between baseline OpenHands (light red) and OpenHands-Perf-Agent (light green). The performance-aware version replaces error reproduction with benchmark creation and emphasizes the high-level sequence of steps we expect the agent to follow to fix a performance bug.
• Figure 5: Performance improvements achieved by OpenHands-Perf-Agent. The horizontal boxplots show the distribution of CPU and Memory improvements in both relative (%) and absolute units ($\mu$s for CPU usage, bytes for allocations) across benchmark tests.