Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

Xue Liu, Xin Ma, Yuxin Ma, Yongchang Peng, Duo Wang, Zhoufutu Wen, Ge Zhang, Kaiyuan Zhang, Xinyu Chen, Tianci He, Jiani Hou, Liang Hu, Ziyun Huang, Yongzhe Hui, Jianpeng Jiao, Chennan Ju, Yingru Kong, Yiran Li, Mengyun Liu, Luyao Ma, Fei Ni, Yiqing Ni, Yueyan Qiu, Yanle Ren, Zilin Shi, Zaiyuan Wang, Wenjie Yue, Shiyu Zhang, Xinyi Zhang, Kaiwen Zhao, Zhenwei Zhu, Shanshan Wu, Qi Zhao, Wenhao Huang

Abstract

As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.

Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

Abstract

As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.

Paper Structure

This paper contains 30 sections, 1 equation, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Expert-level-task
  4. Rubrics evaluated benchmark
  5. Overview of XpertBench
  6. Expert Recruitment and Training
  7. Prompt Curation
  8. Rubric Design and Quality Control
  9. Evaluation
  10. Experiments and Results
  11. Contributions
  12. Xpert Platform
  13. What is Xpert Platform
  14. Xpert Leaderboard Intro
  15. Example Tasks and Scoring Rubrics
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Overview of the benchmark construction and evaluation pipeline, consisting of expert recruitment, task curation, rubric design, and the SHOTJUDGE evaluation framework.
  • Figure 2: Distribution of tasks across professional domains in XpertBench, highlighting the balance between STEM and Humanities fields.
  • Figure 3: Results on XpertBench-Gold evaluation subset (N=245).
  • Figure A.1: The recognition sequence and cleavage sites (indicated by triangles) of the restriction enzymes MboI and Sau3AI. Both enzymes recognize the same $5^\prime$-GATC-$3^\prime$ nucleotide sequence.
  • Figure A.2: Agarose gel electrophoresis results for plasmid verification. Top (Figure 1): Digestion profiles of plasmids extracted from E. coli DH5$\alpha$. Lanes 1--20 are samples treated with MboI; lanes 21--22 (CE) are undigested control plasmids. Bottom (Figure 2): Digestion profiles of the same plasmids after being transferred to and extracted from E. coli ET12567. The left panel shows digestion with MboI, and the right panel shows digestion with Sau3AI. 'M' denotes the DNA molecular weight marker (ladder).