Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li, Xiang Xu, Bohan Wang, Peng Wang, Xingzhe Wu, Anfeng Li, Qiyuan Feng, Yuhao Zhou, Shoulin Han, Wenjie Luo, Yiyuan Li, Yaxuan Wang, Ruixian Luo, Guojie Lin, Peiyao Xiao, Chengliang Xu, Ben Wang, Zeyu Wang, Zichao Chen, Jianan Ye, Yijie Hu, Jialong Chen, Zongwen Shen, Yuliang Xu, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Hu Wei, Que Shen, Bing Zhao

TL;DR

HLE‑Verified systematically audits Humanity’s Last Exam to remove annotation noise that biases model evaluation. By decomposing items into Problem, Rationale, and Answer and applying a two‑stage verification plus revision pipeline, it yields 641 gold, 1,170 revised, and 689 uncertain items, with substantial accuracy and calibration gains across seven LLMs on revised content. The work introduces a 19‑category defect taxonomy and extensive metadata to enable auditable, domain‑sensitive repairs and future refinements. Empirically, verification reduces noise‑driven discrepancies and improves cross‑domain interpretability, offering a path toward more faithful measurement of model capabilities. The release invites ongoing community participation to maintain and extend a reliable benchmark infrastructure for multi‑domain reasoning tasks.

Abstract

Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified

HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

TL;DR

HLE‑Verified systematically audits Humanity’s Last Exam to remove annotation noise that biases model evaluation. By decomposing items into Problem, Rationale, and Answer and applying a two‑stage verification plus revision pipeline, it yields 641 gold, 1,170 revised, and 689 uncertain items, with substantial accuracy and calibration gains across seven LLMs on revised content. The work introduces a 19‑category defect taxonomy and extensive metadata to enable auditable, domain‑sensitive repairs and future refinements. Empirically, verification reduces noise‑driven discrepancies and improves cross‑domain interpretability, offering a path toward more faithful measurement of model capabilities. The release invites ongoing community participation to maintain and extend a reliable benchmark infrastructure for multi‑domain reasoning tasks.

Abstract

Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified
Paper Structure (53 sections, 9 equations, 11 figures, 2 tables)

This paper contains 53 sections, 9 equations, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background
  4. Why benchmark errors matter
  5. HLE as an evaluation substrate
  6. Dataset Verification Process and Methods
  7. Overview of the pipeline
  8. Stage I: component-wise verification
  9. Goal.
  10. Operational definition of validity.
  11. Multi-source verification and adjudication.
  12. Stage I outcome.
  13. Stage II: systematic revision and re-verification
  14. Goal.
  15. Scope and boundary.
  16. ...and 38 more sections

Figures (11)

  • Figure 1: Structural composition of HLE-Verified.
  • Figure 2: HLE Revision Stage I. High-Difficulty Problem Validity Verification & Golden Subset Construction
  • Figure 3: HLE Revision Stage II. Systematic Revision of Challenging Questions
  • Figure 4: HLE Component-wise Defect Taxonomy
  • Figure 5: Overall annotation outcomes on the problematic data of HLE. We report the counts of three labels---valid (1), invalid (0), and uncertain---for each annotatable component: Problem, Answer, and Rationale. The results show clear component-wise differences, with substantially more invalid/uncertain cases in Answer and especially Rationale than in Problem.
  • ...and 6 more figures