IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks

TL;DR

IDE-Bench provides a Dockerized, IDE-native benchmark for evaluating AI agents as IDE collaborators across eight unpublished, real-world repositories spanning multiple languages. By enabling tool-calling through a 17-tool IDE interface and strict evaluation protocols, the framework reveals how models navigate multi-file, multi-language software tasks, beyond single-shot reasoning. Key findings show a production-ready performance threshold around $85\%$ $pass@5$, significant task-level specialization across languages and domains, and diverse efficiency and stability profiles that inform deployment strategies. The work highlights the importance of task-aware routing, iterative learning dynamics, and robust failure analyses to guide practical use and future improvements of AI-powered IDE agents.

Abstract

IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface. We present a Dockerized test harness that goes beyond raw terminal execution, granting models a structured tool ecosystem that represents AI-native IDEs like Cursor and Windsurf. By providing high-level abstractions for codebase search, structured file editing, and tools for testing full-stack applications, IDE-Bench evaluates an agent's ability to act as a true engineering collaborator. For evaluation and to prevent training data contamination, we created 80 tasks across eight never-published repositories spanning C/C++, Java, and MERN stacks, representing modern tech stack production scenarios, including feature implementation, bug fixing, refactoring, and performance optimization that mirror daily developer workflows in private codebases. Our benchmark is the first to systematically correlate agent-reported intent with successful project-level modifications in a multi-language, full-stack environment on completely uncontaminated code.

Figures (27)

• Figure 2: Cumulative distribution function (CDF) of iterations to first successful edit by model. Steeper curves indicate faster initial progress. Only includes runs with at least one successful edit.
• Figure 3: Distribution of total iterations by model. Box shows quartiles (Q1, median, Q3), whiskers extend to 1.5 times IQR, and points show outliers. Lower boxes indicate more efficient completion.
• Figure 4: Scatter plot of total iterations vs. task success (jittered for visibility). Each point represents one run. Successful runs (top) and failed runs (bottom) show distinct iteration patterns.
• Figure 5: Stacked bar chart showing iteration productivity breakdown by model. Green: productive iterations (successful edits), Blue: exploration iterations (no edit attempts), Red: non-productive iterations (failed edits).
• Figure 6: Distribution of failure modes across all failed runs. Note that runs may exhibit multiple failure modes; percentages sum to $>100\%$.