Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations

TL;DR

This paper tackles the mismatch between traditional LLM benchmarks and real-world enterprise agentic AI needs by introducing the Kamiwaza Agentic Merit Index (KAMI) v0.1, a contamination-resistant benchmark built on the PICARD framework. It systematically evaluates 35 model configurations across 5.5B tokens and 170,527 test conversations to measure end-to-end agentic capabilities such as multi-step tool use and reasoning under realistic tasks (CSV, databases, filesystem). Key findings include a persistent agentic disconnect where newer models do not consistently outperform older ones on enterprise tasks, and that aggregation across standard benchmarks remains a weak predictor of practical performance. The work emphasizes the importance of task-specific evaluation, reliability metrics, and context/tool engineering, and proposes a path toward a production-ready, enterprise-relevant benchmarking standard akin to SPEC CPU for agentic AI. It also highlights practical considerations for deployment costs and latency, showing that strategic prompting and tooling can dramatically alter outcomes. The overall contribution is a foundational, scalable framework aimed at guiding enterprise decisions for agentic AI adoption beyond leaderboard-driven assessments.

Abstract

Enterprise adoption of agentic AI systems requires reliable evaluation methods that reflect real-world deployment scenarios. Traditional LLM benchmarks suffer from training data contamination and fail to measure agentic capabilities such as multi-step tool use and decision-making under uncertainty. We present the Kamiwaza Agentic Merit Index (KAMI) v0.1, an enterprise-focused benchmark that addresses both contamination resistance and agentic evaluation. Through 170,000 LLM test items processing over 5.5 billion tokens across 35 model configurations, we demonstrate that traditional benchmark rankings poorly predict practical agentic performance. Notably, newer generation models like Llama 4 or Qwen 3 do not always outperform their older generation variants on enterprise-relevant tasks, contradicting traditional benchmark trends. We also present insights on cost-performance tradeoffs, model-specific behavioral patterns, and the impact of reasoning capabilities on token efficiency -- findings critical for enterprises making deployment decisions.