Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Programming with Pixels: Can Computer-Use Agents do Software Engineering?

Pranjal Aggarwal, Sean Welleck

TL;DR

This work evaluates whether generalist computer-use agents can perform software engineering tasks using Programming with Pixels (PwP), an IDE-based environment paired with PwP-Bench, a 15-task, multimodal SWE benchmark. Pure visual CUAs perform poorly (approx. 23% accuracy), but granting two text APIs (file editing and bash) boosts performance to around 51%, with further gains when IDE tools are accessible via text prompts. The study identifies critical limitations in visual grounding and tool utilization, while also showing rapid progress over time for advanced models and the potential of assisted tool use to close the gap with specialized SWE agents. PwP provides a unified, open platform to benchmark and drive improvements in generalist agents for complex software-engineering tasks, highlighting the need for better grounding, planning, and IDE-tool integration.

Abstract

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce $\texttt{Programming with Pixels}$ (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at https://programmingwithpixels.com

Programming with Pixels: Can Computer-Use Agents do Software Engineering?

TL;DR

This work evaluates whether generalist computer-use agents can perform software engineering tasks using Programming with Pixels (PwP), an IDE-based environment paired with PwP-Bench, a 15-task, multimodal SWE benchmark. Pure visual CUAs perform poorly (approx. 23% accuracy), but granting two text APIs (file editing and bash) boosts performance to around 51%, with further gains when IDE tools are accessible via text prompts. The study identifies critical limitations in visual grounding and tool utilization, while also showing rapid progress over time for advanced models and the potential of assisted tool use to close the gap with specialized SWE agents. PwP provides a unified, open platform to benchmark and drive improvements in generalist agents for complex software-engineering tasks, highlighting the need for better grounding, planning, and IDE-tool integration.

Abstract

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at https://programmingwithpixels.com

Paper Structure

This paper contains 48 sections, 16 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Programming with Pixels (PwP)
  4. PwP environment.
  5. Infrastructure and Implementation
  6. PwP-Bench
  7. Tasks.
  8. Benchmarking Design and Task Setup.
  9. Evaluating Agents in Programming with Pixels
  10. Experimental setup.
  11. Results and Analysis
  12. Computer-Use Agents Demonstrate Poor Visual Grounding Capabilities.
  13. Computer-Use Agents Are Rapidly Improving.
  14. Conclusion
  15. Acknowledgements
  16. ...and 33 more sections

Figures (16)

  • Figure 1: Programming with Pixels is an environment for computer-use agents, where they interact with a VSCode IDE through keyboard and mouse actions while observing the screen. The framework supports multiple programming languages, tests interactions with multiple IDE features, modalities (eg: text, images, data files). PwP-Bench evaluates agents across 15 diverse software engineering tasks such as code generation, UI generation, Data Science.
  • Figure 2: Example of Successful Use of Live Preview Tool in the UI Replication Task The agent successfully uses the live preview tool in the VSCode browser to compare the UI design it made versus the reference design.
  • Figure 3: Example of Successful Use of Tool in the Chart Generation Task The agent can compare the generated chart with the reference chart side by side and refine its code accordingly.
  • Figure 4: Agent Hallucinating Screen Contents The agent correctly mentions, search results are displayed (green), it hallucinates an input field containing "disable import error" (red).
  • Figure 5: Agent Misidentifying UI Elements The agent fails to identify the correct input field, typing '50' into the settings search bar instead of the word wrap column setting field (red arrow).
  • ...and 11 more figures