Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Agents in Insurance Underwriting Environments

Amanda Dsouza, Ramya Ramakrishnan, Charles Dickens, Bhavishya Pohani, Christopher M Glaze

TL;DR

Underwrite introduces an expert-first, multi-turn insurance underwriting benchmark that captures enterprise realism, including proprietary knowledge, noisy data, and imperfect user behavior. The environment combines a copilot-like underwriting system, a Model Context Protocol for tool access, and a simulated underwriter to evaluate 13 frontier agents across 3000 synthetic applicant scenarios. Key findings show that expert involvement improves realism, agent frameworks exhibit brittleness, and compositional approaches are needed for reliable hallucination detection in complex domains. The work provides principles for building enterprise-ready benchmarks and helps bridge the gap between lab performance and deployment readiness in specialized settings.

Abstract

As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness: the most accurate models are not the most efficient, models hallucinate domain knowledge despite tool access, and pass^k results show a 20% drop in performance. The results from UNDERWRITE demonstrate that expert involvement in benchmark design is essential for realistic agent evaluation, common agentic frameworks exhibit brittleness that skews performance reporting, and hallucination detection in specialized domains demands compositional approaches. Our work provides insights for developing benchmarks that better align with enterprise deployment requirements.

Benchmarking Agents in Insurance Underwriting Environments

TL;DR

Underwrite introduces an expert-first, multi-turn insurance underwriting benchmark that captures enterprise realism, including proprietary knowledge, noisy data, and imperfect user behavior. The environment combines a copilot-like underwriting system, a Model Context Protocol for tool access, and a simulated underwriter to evaluate 13 frontier agents across 3000 synthetic applicant scenarios. Key findings show that expert involvement improves realism, agent frameworks exhibit brittleness, and compositional approaches are needed for reliable hallucination detection in complex domains. The work provides principles for building enterprise-ready benchmarks and helps bridge the gap between lab performance and deployment readiness in specialized settings.

Abstract

As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness: the most accurate models are not the most efficient, models hallucinate domain knowledge despite tool access, and pass^k results show a 20% drop in performance. The results from UNDERWRITE demonstrate that expert involvement in benchmark design is essential for realistic agent evaluation, common agentic frameworks exhibit brittleness that skews performance reporting, and hallucination detection in specialized domains demands compositional approaches. Our work provides insights for developing benchmarks that better align with enterprise deployment requirements.
Paper Structure (45 sections, 12 figures, 3 tables)

This paper contains 45 sections, 12 figures, 3 tables.

Table of Contents

  1. Appendix
  2. The Underwrite Environment
  3. Underwriting System
  4. Data and Tool Resources
  5. Data
  6. Tools
  7. Task Definitions
  8. Seed Task Types
  9. Task Diversity
  10. Task Complexity
  11. Agent and User Models
  12. Agent Framework
  13. Simulated User
  14. Rewards
  15. Key Findings from Developing Underwrite
  16. ...and 30 more sections

Figures (12)

  • Figure 1: Architecture of the system
  • Figure 2: Representation of our expert engagement process: As in other approaches to benchmark development, experts reviewed and curated individual tasks (top diagram). However, they also collaborated as a group in developing the system more holistically as a representation of a product backend, with realistic business rules and tables (bottom diagram).
  • Figure 3: pass^ k measured on Answer Correctness across all task types for GPT-5 and Claude-Sonnet-4.5.
  • Figure 4: Failure rates of agents, aggregated over all completed traces, and models. Tool error rate measures the % of traces with at least one tool error. Similarly, uncertain response rate and answer hallucination rate measures the % of traces with at least one uncertain response, and one hallucinated answer, respectively.
  • Figure 5: Completed traces of incorrect outcomes, across all models, tend to use fewer steps, but a higher number of tokens, indicating more verbose responses, as compared to those of correct outcomes.
  • ...and 7 more figures