Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments

TL;DR

This paper introduces EnterpriseBench, a comprehensive benchmark and sandbox for evaluating LLM-based agents in realistic enterprise environments characterized by data fragmentation and strict access controls. It couples a data-grounded sandbox with an automated task-generation pipeline to produce 500 domain-spanning tasks and a suite of enterprise tools, enabling end-to-end evaluation of planning and action execution. Across multiple models and planning strategies, experiments reveal substantial gaps between current CAI capabilities and human performance, with best-performing agents achieving roughly 41–52% task completion and humans achieving ~70% but with longer turnaround. The work highlights critical challenges in enterprise AI—from tool selection and context retrieval to grounding and access control—and provides a foundation for developing more robust, enterprise-ready AI systems and benchmarks.

Abstract

Enterprise systems are crucial for enhancing productivity and decision-making among employees and customers. Integrating LLM based systems into enterprise systems enables intelligent automation, personalized experiences, and efficient information retrieval, driving operational efficiency and strategic growth. However, developing and evaluating such systems is challenging due to the inherent complexity of enterprise environments, where data is fragmented across multiple sources and governed by sophisticated access controls. We present EnterpriseBench, a comprehensive benchmark that simulates enterprise settings, featuring 500 diverse tasks across software engineering, HR, finance, and administrative domains. Our benchmark uniquely captures key enterprise characteristics including data source fragmentation, access control hierarchies, and cross-functional workflows. Additionally, we provide a novel data generation pipeline that creates internally consistent enterprise tasks from organizational metadata. Experiments with state-of-the-art LLM agents demonstrate that even the most capable models achieve only 41.8% task completion, highlighting significant opportunities for improvement in enterprise-focused AI systems.

Figures (10)

• Figure 1: Task Execution in EnterpriseBench. This figure illustrates how an LLM-based agent interacts with the enterprise environment. Given a task, the agent perceives the available enterprise tools, applications, and data sources, formulates a reasoning plan, and executes actions to complete the task.
• Figure 2: Classification of Tasks by Domain (counts)
• Figure 3: Comparison of different models using ReAct planning: Performance across different domains of EnterpriseBench.
• Figure 4: Domain Expert Validation in EnterpriseBench. Domain experts from all benchmark domains evaluate the realism of the generated data and created tasks. This example shows screenshots of MS form for different steps a domain expert completes during the validation process.
• Figure 5: Expert-curated ER diagram for the EnterpriseBench sandbox