Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems

TL;DR

The paper tackles the gap between accuracy-centric agent benchmarks and real-world enterprise deployment needs. It introduces the CLEAR framework, a five-dimension evaluation (Cost, Latency, Efficacy, Assurance, Reliability) with metrics such as CNA, CPS, PAS, and pass@k, enabling multi-objective optimization for production readiness. An Enterprise Task Suite of 300 tasks across six domains evaluates six agents, showing that optimizing for accuracy alone incurs $4.4$–$10.8\times$ higher costs with little extra efficacy, and that domain-tuned approaches can achieve superior cost-normalized performance and reliability. Expert validation demonstrates that CLEAR better predicts deployment success (\(\rho=0.83\)) than accuracy-only evaluation (\(\rho=0.41\)), underscoring the practical value of multidimensional enterprise evaluation for agentic AI systems.

Abstract

Current agentic AI benchmarks predominantly evaluate task completion accuracy, while overlooking critical enterprise requirements such as cost-efficiency, reliability, and operational stability. Through systematic analysis of 12 main benchmarks and empirical evaluation of state-of-the-art agents, we identify three fundamental limitations: (1) absence of cost-controlled evaluation leading to 50x cost variations for similar precision, (2) inadequate reliability assessment where agent performance drops from 60\% (single run) to 25\% (8-run consistency), and (3) missing multidimensional metrics for security, latency, and policy compliance. We propose \textbf{CLEAR} (Cost, Latency, Efficacy, Assurance, Reliability), a holistic evaluation framework specifically designed for enterprise deployment. Evaluation of six leading agents on 300 enterprise tasks demonstrates that optimizing for accuracy alone yields agents 4.4-10.8x more expensive than cost-aware alternatives with comparable performance. Expert evaluation (N=15) confirms that CLEAR better predicts production success (correlation $ρ=0.83$) compared to accuracy-only evaluation ($ρ=0.41$).