Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, Chen Sun, Han Hou, Hui Yang, James Pan, Jianan Lou, Jiayi Mao, Jizheng Liu, Jinpeng Li, Kangyi Liu, Kenkun Liu, Rui Wang, Run Li, Tong Niu, Wenlong Zhang, Wenqi Yan, Xuanzheng Wang, Yuchen Zhang, Yi-Hsin Hung, Yuan Jiang, Zexuan Liu, Zihan Yin, Zijian Ma, Zhiwen Mo

TL;DR

xbench introduces a profession-aligned evaluation paradigm that ties AI agent productivity to real-world, domain-specific workflows. By focusing on live, expert-defined tasks in Recruitment and Marketing and employing LLM-based rubrics alongside IRT-based capability tracking, the framework aims to predict Tech-Market Fit and monitor long-term capability growth. Early results establish baselines and reveal market dynamics, while the open, continuously updated design supports ongoing evaluation of domain-specific agents. The approach promises a value-centric path for advancing AI agents under real-world economic constraints and competition analysis.

Abstract

We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.

xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

TL;DR

xbench introduces a profession-aligned evaluation paradigm that ties AI agent productivity to real-world, domain-specific workflows. By focusing on live, expert-defined tasks in Recruitment and Marketing and employing LLM-based rubrics alongside IRT-based capability tracking, the framework aims to predict Tech-Market Fit and monitor long-term capability growth. Early results establish baselines and reveal market dynamics, while the open, continuously updated design supports ongoing evaluation of domain-specific agents. The approach promises a value-centric path for advancing AI agents under real-world economic constraints and competition analysis.

Abstract

We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.

Paper Structure

This paper contains 27 sections, 1 equation, 9 figures, 19 tables.

Table of Contents

  1. Introduction
  2. AI Capability Centric v.s. Profession-Aligned
  3. Building xbench
  4. Live Professional Demands Define Evaluation Tasks
  5. Benchmarking Recruitment
  6. Task Design and Examples
  7. Evaluation Process
  8. Task Distribution
  9. Benchmarking Marketing
  10. Task Design and Examples
  11. Evaluation Process
  12. Task Distribution
  13. Evalutions
  14. Setups
  15. Benchmarks
  16. ...and 12 more sections

Figures (9)

  • Figure 1: Profession-aligned evaluation define domain agents, predict Tech-Market Fit (TMF) and track competition of agent products.
  • Figure 2: Difference between AI-capability-centric and profession-aligned benchmarks
  • Figure 3: Evaluation pipelines for recruitment tasks.
  • Figure 4: Task distribution across recruitment tasks' themes and human time cost (in minutes).
  • Figure 5: The evaluation pipeline for the Influencer Search task in marketing benchmark.
  • ...and 4 more figures