Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELAIPBench: A Benchmark for Expert-Level Artificial Intelligence Paper Understanding

Xinbang Dai, Huikang Hu, Yongrui Chen, Jiaqi Li, Rihui Jin, Yuyang Zhang, Xiaoguang Li, Lifeng Shang, Guilin Qi

TL;DR

ELAIPBench targets expert-level understanding of AI papers by evaluating LLMs on 403 MCQs drawn from 137 papers, using a competitive, incentive-driven annotation process to ensure high-quality, reasoning-dependent questions. The benchmark reveals that even frontier LLMs struggle to achieve high accuracy (best about 40%), with reasoning modes and RAG often harming performance due to overthinking and noisy retrieval. Findings include robust evidence that longer reasoning does not guarantee better results, and that harmful verification is a major failure mode in current systems. The work highlights a clear gap between current LLM capabilities and genuine comprehension of scholarly texts, underscoring the need for more sophisticated evidence-grounded reasoning and retrieval strategies in AI research tools.

Abstract

While large language models (LLMs) excel at many domain-specific tasks, their ability to deeply comprehend and reason about full-length academic papers remains underexplored. Existing benchmarks often fall short of capturing such depth, either due to surface-level question design or unreliable evaluation metrics. To address this gap, we introduce ELAIPBench, a benchmark curated by domain experts to evaluate LLMs' comprehension of artificial intelligence (AI) research papers. Developed through an incentive-driven, adversarial annotation process, ELAIPBench features 403 multiple-choice questions from 137 papers. It spans three difficulty levels and emphasizes non-trivial reasoning rather than shallow retrieval. Our experiments show that the best-performing LLM achieves an accuracy of only 39.95%, far below human performance. Moreover, we observe that frontier LLMs equipped with a thinking mode or a retrieval-augmented generation (RAG) system fail to improve final results-even harming accuracy due to overthinking or noisy retrieval. These findings underscore the significant gap between current LLM capabilities and genuine comprehension of academic papers.

ELAIPBench: A Benchmark for Expert-Level Artificial Intelligence Paper Understanding

TL;DR

ELAIPBench targets expert-level understanding of AI papers by evaluating LLMs on 403 MCQs drawn from 137 papers, using a competitive, incentive-driven annotation process to ensure high-quality, reasoning-dependent questions. The benchmark reveals that even frontier LLMs struggle to achieve high accuracy (best about 40%), with reasoning modes and RAG often harming performance due to overthinking and noisy retrieval. Findings include robust evidence that longer reasoning does not guarantee better results, and that harmful verification is a major failure mode in current systems. The work highlights a clear gap between current LLM capabilities and genuine comprehension of scholarly texts, underscoring the need for more sophisticated evidence-grounded reasoning and retrieval strategies in AI research tools.

Abstract

While large language models (LLMs) excel at many domain-specific tasks, their ability to deeply comprehend and reason about full-length academic papers remains underexplored. Existing benchmarks often fall short of capturing such depth, either due to surface-level question design or unreliable evaluation metrics. To address this gap, we introduce ELAIPBench, a benchmark curated by domain experts to evaluate LLMs' comprehension of artificial intelligence (AI) research papers. Developed through an incentive-driven, adversarial annotation process, ELAIPBench features 403 multiple-choice questions from 137 papers. It spans three difficulty levels and emphasizes non-trivial reasoning rather than shallow retrieval. Our experiments show that the best-performing LLM achieves an accuracy of only 39.95%, far below human performance. Moreover, we observe that frontier LLMs equipped with a thinking mode or a retrieval-augmented generation (RAG) system fail to improve final results-even harming accuracy due to overthinking or noisy retrieval. These findings underscore the significant gap between current LLM capabilities and genuine comprehension of academic papers.

Paper Structure

This paper contains 54 sections, 20 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Scientific Comprehension Benchmarks
  4. Academic Benchmark Annotation Methods
  5. Construction of ElaipBench
  6. Task Overview
  7. Task Definition.
  8. Annotation Team.
  9. Competitive Incentive Annotation Mechanism
  10. QA Pair Generation
  11. Evidence Verification
  12. Answer Verification
  13. Benchmark Statistics
  14. Experiments
  15. Baseline LLMs
  16. ...and 39 more sections

Figures (20)

  • Figure 1: An example from ElaipBench. Option A, while potentially factually correct, is not substantiated by the text. Option C is a deliberately constructed distractor with no direct supporting evidence in the text. Option D is also not explicitly stated in the text, although its latter part is derived from the paper.
  • Figure 2: Construction of ElaipBench.
  • Figure 3: Reasoning error distribution of ElaipBnech on different LLMs.
  • Figure 4: Completion Tokens vs.Performance.
  • Figure 5: The reasoning lengths across different question types (SE: SA-MCQ + easy, SM: SA-MCQ + moderate, SH: SA-MCQ + hard, ME: MA-MCQ + easy, MM: MA-MCQ + moderate, MH: MA-MCQ + hard).
  • ...and 15 more figures