Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

World of Workflows: a Benchmark for Bringing World Models to Enterprise Systems

Lakshya Gupta, Litao Li, Yizhe Liu, Sriram Ganapathi Subramanian, Kaheer Suleman, Zichen Zhang, Haoye Lu, Sumit Pasupalak

TL;DR

The World of Workflows (WoW) paper introduces a realistic ServiceNow-based enterprise environment and WoW-bench, a 234-task benchmark designed to stress-test frontier LLMs as enterprise world models and agents. It demonstrates that current models suffer from dynamics blindness in partially observable, workflow-driven settings and that reliability improves only when agents receive grounded state evidence via table audits, highlighting the need for dynamics-aware reasoning. The work provides a formal POMDP formulation, two observation modes, a tool-dependency graph trajectory sampler, and extensive evaluation across constraint understanding, autonomous task completion, and dynamics/prediction tasks, revealing major gaps in symbolic grounding, transition modeling, and multi-hop causality. Overall, WoW shifts the focus from surface-level task completion to faithful modeling of hidden system dynamics, offering a path toward robust, dynamics-aware enterprise agents with practical implications for real-world organizational AI applications.

Abstract

Frontier large language models (LLMs) excel as autonomous agents in many domains, yet they remain untested in complex enterprise systems where hidden workflows create cascading effects across interconnected databases. Existing enterprise benchmarks evaluate surface-level agentic task completion similar to general consumer benchmarks, ignoring true challenges in enterprises, such as limited observability, large database state, and hidden workflows with cascading side effects. We introduce World of Workflows (WoW), a realistic ServiceNow-based environment incorporating 4,000+ business rules and 55 active workflows embedded in the system, alongside WoW-bench, a benchmark of 234 tasks evaluating constrained agentic task completion and enterprise dynamics modeling capabilities. We reveal two major takeaways: (1) Frontier LLMs suffer from dynamics blindness, consistently failing to predict the invisible, cascading side effects of their actions, which leads to silent constraint violations, and (2) reliability in opaque systems requires grounded world modeling, where agents must mentally simulate hidden state transitions to bridge the observability gap when high-fidelity feedback is unavailable. For reliable and useful enterprise agents, WoW motivates a new paradigm to explicitly learn system dynamics. We release our GitHub for setting up and evaluating WoW.

World of Workflows: a Benchmark for Bringing World Models to Enterprise Systems

TL;DR

The World of Workflows (WoW) paper introduces a realistic ServiceNow-based enterprise environment and WoW-bench, a 234-task benchmark designed to stress-test frontier LLMs as enterprise world models and agents. It demonstrates that current models suffer from dynamics blindness in partially observable, workflow-driven settings and that reliability improves only when agents receive grounded state evidence via table audits, highlighting the need for dynamics-aware reasoning. The work provides a formal POMDP formulation, two observation modes, a tool-dependency graph trajectory sampler, and extensive evaluation across constraint understanding, autonomous task completion, and dynamics/prediction tasks, revealing major gaps in symbolic grounding, transition modeling, and multi-hop causality. Overall, WoW shifts the focus from surface-level task completion to faithful modeling of hidden system dynamics, offering a path toward robust, dynamics-aware enterprise agents with practical implications for real-world organizational AI applications.

Abstract

Frontier large language models (LLMs) excel as autonomous agents in many domains, yet they remain untested in complex enterprise systems where hidden workflows create cascading effects across interconnected databases. Existing enterprise benchmarks evaluate surface-level agentic task completion similar to general consumer benchmarks, ignoring true challenges in enterprises, such as limited observability, large database state, and hidden workflows with cascading side effects. We introduce World of Workflows (WoW), a realistic ServiceNow-based environment incorporating 4,000+ business rules and 55 active workflows embedded in the system, alongside WoW-bench, a benchmark of 234 tasks evaluating constrained agentic task completion and enterprise dynamics modeling capabilities. We reveal two major takeaways: (1) Frontier LLMs suffer from dynamics blindness, consistently failing to predict the invisible, cascading side effects of their actions, which leads to silent constraint violations, and (2) reliability in opaque systems requires grounded world modeling, where agents must mentally simulate hidden state transitions to bridge the observability gap when high-fidelity feedback is unavailable. For reliable and useful enterprise agents, WoW motivates a new paradigm to explicitly learn system dynamics. We release our GitHub for setting up and evaluating WoW.
Paper Structure (38 sections, 7 equations, 6 figures, 7 tables)

This paper contains 38 sections, 7 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. World Models
  4. General Agentic Benchmarks
  5. Enterprise Benchmarks
  6. WoW Environment
  7. Environment Specification
  8. 1. Standard Observation (Tool Response)
  9. 2. Oracle Observation (Table Audits)
  10. Workflows
  11. WoW-bench
  12. Task Categories
  13. Constraint Understanding
  14. Agentic Task Completion
  15. Action and Audit Prediction
  16. ...and 23 more sections

Figures (6)

  • Figure 1: Illustration of Cascading Workflow Failures in WoW. This diagram demonstrates how a single agent action can trigger a sequence of hidden state changes that violate constraints.
  • Figure 2: WoW vs. previous enterprise benchmarks. WoW includes complex workflows and business rules, which introduce hidden data flow and API calls that can impact the database state. Compared to previous environments, which do not involve workflows, WoW requires agents to model the system dynamics to correctly carry out tasks and follow constraints. We introduce an augmentation to the observation space with table audits as tractable state changes to provide low-level information to agents.
  • Figure 3: Aggregated results for all task categories with frontier LLMs.
  • Figure 4: Tool-based dependency graph
  • Figure 5: Performance of frontier LLMs across task categories. Constraint understanding and agentic tasks are evaluated using two observation modes: tool response $\mathcal{O}_{tool}$ and table audits $\mathcal{O}_{audit}$.
  • ...and 1 more figures