ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments

TL;DR

ProSoftArena tackles the gap in evaluating multimodal agents within professional software by introducing a hierarchical capability taxonomy and a VM-based, execution-focused benchmark spanning 436 tasks across 6 disciplines and 13 applications. It combines automated task initialization, execution-based scoring, and a human-in-the-loop paradigm to measure collaborative efficiency and reliability in real-world software environments. Experimental results reveal strong performance gaps: L1 is hard, L2 is the primary bottleneck, and cross-application L3 workflows remain largely unattainable; human collaboration and richer multimodal observations provide meaningful gains but expose trade-offs in cost and complexity. By delivering a scalable, reproducible testbed and actionable insights on planning, grounding, and domain knowledge, ProSoftArena offers a concrete roadmap for advancing professional AI agents toward higher levels of autonomy and collaboration.

Abstract

Multimodal agents are making rapid progress on general computer-use tasks, yet existing benchmarks remain largely confined to browsers and basic desktop applications, falling short in professional software workflows that dominate real-world scientific and industrial practice. To close this gap, we introduce ProSoftArena, a benchmark and platform specifically for evaluating multimodal agents in professional software environments. We establish the first capability hierarchy tailored to agent use of professional software and construct a benchmark of 436 realistic work and research tasks spanning 6 disciplines and 13 core professional applications. To ensure reliable and reproducible assessment, we build an executable real-computer environment with an execution-based evaluation framework and uniquely incorporate a human-in-the-loop evaluation paradigm. Extensive experiments show that even the best-performing agent attains only a 24.4\% success rate on L2 tasks and completely fails on L3 multi-software workflow. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents in professional software settings. This project is available at: https://prosoftarena.github.io.

Figures (14)

• Figure 1: We establish the first hierarchical taxonomy of agent capabilities in professional software environments; and curate a comprehensive benchmark covering 6 disciplines, 20 subfields and 13 core professional applications. We construct a VM-based real computer environment for reproducible evaluations, and uniquely incorporate a human-in-the-loop evaluation paradigm.
• Figure 2: Representative Task Samples across Six Core Domains in ProSoftArena. For each domain, we showcase a typical task scenario including the specific natural language instruction, the initial computer state, and associated metadata.
• Figure 3: Automated and Human-in-the-loop Evaluation Framework.
• Figure 4: Qualitative Examples of Agent Trajectories. The top row shows the process of adding a north arrow in ArcGIS, involving view switching and dialog interaction. The bottom row illustrates adding a simulation result in ANSYS through tree navigation and context menus. Red dashed lines and blue cursors indicate mouse movements and clicks, respectively.
• Figure 5: Model performance across applications. Left: L1 tasks. Right: L2 tasks. Both under Screenshot+A11y observations.