Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension

Yelin Chen, Fanjin Zhang, Suping Sun, Yunhe Pang, Yuanchun Wang, Jian Song, Xiaoyan Li, Lei Hou, Shu Zhao, Jie Tang, Juanzi Li

TL;DR

RPC-Bench tackles the challenge of evaluating deep understanding of research papers by building a large-scale, authentic QA benchmark from OpenReview review–rebuttal exchanges. It introduces a fine-grained taxonomy spanning Concepts, Methods, Experiments, and Claim Verification, and a scalable LLM–human annotation framework with a rigorous evaluation protocol that measures correctness, completeness, and conciseness. The study analyzes 28 models across pure-text and multimodal inputs, revealing that no model fully comprehends scholarly content and that multimodal grounding remains a major bottleneck. By providing ground-truth QA grounded in original papers and aligning automated judgments with human evaluations, RPC-Bench offers a realistic, architecture-agnostic platform for advancing research on machine comprehension of scientific literature and multimodal scholarly documents.

Abstract

Understanding research papers remains challenging for foundation models due to specialized scientific discourse and complex figures and tables, yet existing benchmarks offer limited fine-grained evaluation at scale. To address this gap, we introduce RPC-Bench, a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers, containing 15K human-verified QA pairs. We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts. We also define an elaborate LLM-human interaction annotation framework to support large-scale labeling and quality control. Following the LLM-as-a-Judge paradigm, we develop a scalable framework that evaluates models on correctness-completeness and conciseness, with high agreement to human judgment. Experiments reveal that even the strongest models (GPT-5) achieve only 68.2% correctness-completeness, dropping to 37.46% after conciseness adjustment, highlighting substantial gaps in precise academic paper understanding. Our code and data are available at https://rpc-bench.github.io/.

RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension

TL;DR

RPC-Bench tackles the challenge of evaluating deep understanding of research papers by building a large-scale, authentic QA benchmark from OpenReview review–rebuttal exchanges. It introduces a fine-grained taxonomy spanning Concepts, Methods, Experiments, and Claim Verification, and a scalable LLM–human annotation framework with a rigorous evaluation protocol that measures correctness, completeness, and conciseness. The study analyzes 28 models across pure-text and multimodal inputs, revealing that no model fully comprehends scholarly content and that multimodal grounding remains a major bottleneck. By providing ground-truth QA grounded in original papers and aligning automated judgments with human evaluations, RPC-Bench offers a realistic, architecture-agnostic platform for advancing research on machine comprehension of scientific literature and multimodal scholarly documents.

Abstract

Understanding research papers remains challenging for foundation models due to specialized scientific discourse and complex figures and tables, yet existing benchmarks offer limited fine-grained evaluation at scale. To address this gap, we introduce RPC-Bench, a large-scale question-answering benchmark built from review-rebuttal exchanges of high-quality computer science papers, containing 15K human-verified QA pairs. We design a fine-grained taxonomy aligned with the scientific research flow to assess models' ability to understand and answer why, what, and how questions in scholarly contexts. We also define an elaborate LLM-human interaction annotation framework to support large-scale labeling and quality control. Following the LLM-as-a-Judge paradigm, we develop a scalable framework that evaluates models on correctness-completeness and conciseness, with high agreement to human judgment. Experiments reveal that even the strongest models (GPT-5) achieve only 68.2% correctness-completeness, dropping to 37.46% after conciseness adjustment, highlighting substantial gaps in precise academic paper understanding. Our code and data are available at https://rpc-bench.github.io/.
Paper Structure (47 sections, 3 equations, 13 figures, 8 tables)

This paper contains 47 sections, 3 equations, 13 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodologies for Document Question Answering.
  4. Document QA Benchmarks.
  5. RPC-Bench
  6. Data Collection
  7. Taxonomy Design
  8. Annotation Process
  9. Quality Control
  10. Evaluation Protocol
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Performance across Task Categories
  15. Performance of Detailed Question Types.
  16. ...and 32 more sections

Figures (13)

  • Figure 1: RPC-Bench Construction Pipeline. We crawl papers and review–rebuttal pairs from OpenReview and apply impact‑aware sampling to balance quality and mitigate bias. Review-rebuttals are segmented into comment–response units with GPT‑4o, rewritten into QA pairs using GLM‑4‑Plus and DeepSeek‑V3. Low‑quality QA items are discarded before iterative human annotation and review.
  • Figure 2: Domain distribution of RPC-Bench. ML: Machine Learning; CV: Computer Vision; NLP: Natural Language Processing; RL: Reinforcement Learning.
  • Figure 3: Task taxonomy of QA pairs. The form of [What-4.27%] indicates question types and QA percentage.
  • Figure 4: Comparison of LLMs and VLMs on open‑ended question answering (F1‑like score; left), and the performance of all models on claim verification tasks (ACC; right).
  • Figure 5: Representative case studies from the RPC‑Bench test set
  • ...and 8 more figures