EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents

TL;DR

EntWorld introduces a scalable, verifiable benchmark for enterprise GUI agents, covering 1,756 tasks across six domains (CRM, ERP, ITSM, and related systems). The environment uses a schema-grounded task generation pipeline that reverse-engineers business logic from underlying database schemas and a deterministic SQL-based verification protocol to validate state transitions, avoiding ambiguous visual judgments. Formalized as an MDP, with a challenge of long-horizon workflows, EntWorld reveals a pronounced enterprise gap: state-of-the-art models such as GPT-4.1 achieve around $SR\approx 0.4761$ while human performance is substantially higher (approximately $SR\approx 0.85$). The authors release a dockerized sandbox and standardized task formats to accelerate progress toward enterprise-ready digital agents, and show that an RL-trained EntAgent-RL (based on Qwen2.5-VL-32B-Instruct) currently reaches state-of-the-art open-source performance at $SR=56.89\%$, highlighting the practical impact of domain-specific agents.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled agents to operate in open-ended web and operating system environments. However, existing benchmarks predominantly target consumer-oriented scenarios (e.g., e-commerce and travel booking), failing to capture the complexity and rigor of professional enterprise workflows. Enterprise systems pose distinct challenges, including high-density user interfaces, strict business logic constraints, and a strong reliance on precise, state-consistent information retrieval-settings in which current generalist agents often struggle. To address this gap, we introduce EntWorld, a large-scale benchmark consisting of 1,756 tasks across six representative enterprise domains, including customer relationship management (CRM), information technology infrastructure library (ITIL), and enterprise resource planning (ERP) systems. Unlike previous datasets that depend on fragile execution traces or extensive manual annotation, EntWorld adopts a schema-grounded task generation framework that directly reverse-engineers business logic from underlying database schemas, enabling the synthesis of realistic, long-horizon workflows. Moreover, we propose a SQL-based deterministic verification mechanism in building datasets that replaces ambiguous visual matching with rigorous state-transition validation. Experimental results demonstrate that state-of-the-art models (e.g., GPT-4.1) achieve 47.61% success rate on EntWorld, substantially lower than the human performance, highlighting a pronounced enterprise gap in current agentic capabilities and the necessity of developing domain-specific agents. We release EntWorld as a rigorous testbed to facilitate the development and evaluation of the next generation of enterprise-ready digital agents.

Figures (9)

• Figure 1: Overview of the EntWorld construction pipeline. The construction process consists of workflow discovery, verifiable task generation, benchmark construction, and quantitative difficulty analysis. Our benchmark EntWorld environment can run in parallel on a single host machine, enabling efficient learning and evaluation.
• Figure 2: Statistics and example of the EntWorld dataset. (\ref{['fig:benckmark_distribute']}) shows the distribution of task intents in EntWorld across domians, with numbers indicating the count of sub-domains. (\ref{['fig:benchmark_example']}) illustrates a sample data instance of our dataset with the webpage screenshot and actions in accessibility tree, where actions in yellow will result in a transition to a new webpage.
• Figure 3: Environment-Specific Analysis. Success Rate (%) across six distinct enterprise environments.
• Figure 4: A case of agent failure, along with their screenshot and the accessibility tree of the relevant sections. The agent correctly interpreted the task instructions, but ultimately produced an incorrect answer due to inaccurate perception of page state changes and interface prompts. The case is about EspoCRM.
• Figure 5: A case of agent failure due to UI element type misidentification, where the agent incorrectly interprets interface components and performs actions inconsistent with the actual environmental state, ultimately leading to deviation from the intended workflow. The case is about iTOP.