Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, Lu Wang

TL;DR

ExpertLongBench introduces a multi-domain, expert-level benchmark with 11 long-form tasks (1050 samples) that reflect real-world workflows and require outputs exceeding thousands of tokens. It pairs each task with expert-crafted rubrics and ground-truth checklist-mapped references, and proposes CLEAR, a grounded, checklist-based evaluation framework that maps model and reference outputs to per-item criteria. In extensive experiments with 13 LLMs, current models struggle on end-to-end expert tasks (average F1 around 33.4) even as some content aligns with checklist aspects, highlighting a gap between surface conformity and correctness. The paper demonstrates that open-weight components can substitute proprietary judges for scalable, reproducible evaluation and analyzes task difficulty and skill decomposition to guide future benchmarking and model development. Overall, ExpertLongBench and CLEAR offer a rigorous, low-cost path toward evaluating and improving expert-level long-form generation in real-world workflows.

Abstract

This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items of model outputs are then compared with corresponding items of reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 13 popular large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer Gemini-2.5-Pro achieving only a 33.4 F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, but far from correct; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable, reproducible, and low-cost usage.

ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

TL;DR

ExpertLongBench introduces a multi-domain, expert-level benchmark with 11 long-form tasks (1050 samples) that reflect real-world workflows and require outputs exceeding thousands of tokens. It pairs each task with expert-crafted rubrics and ground-truth checklist-mapped references, and proposes CLEAR, a grounded, checklist-based evaluation framework that maps model and reference outputs to per-item criteria. In extensive experiments with 13 LLMs, current models struggle on end-to-end expert tasks (average F1 around 33.4) even as some content aligns with checklist aspects, highlighting a gap between surface conformity and correctness. The paper demonstrates that open-weight components can substitute proprietary judges for scalable, reproducible evaluation and analyzes task difficulty and skill decomposition to guide future benchmarking and model development. Overall, ExpertLongBench and CLEAR offer a rigorous, low-cost path toward evaluating and improving expert-level long-form generation in real-world workflows.

Abstract

This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items of model outputs are then compared with corresponding items of reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 13 popular large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer Gemini-2.5-Pro achieving only a 33.4 F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, but far from correct; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable, reproducible, and low-cost usage.

Paper Structure

This paper contains 144 sections, 11 figures, 63 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating LLMs on Expert-Level Knowledge.
  4. Evaluating Long-Form Generations.
  5. Benchmark Construction with Multi-disciplinary Expert Tasks
  6. Overview.
  7. Task Sources.
  8. Expert-guided Rubric Design.
  9. Sample Selection.
  10. Checklist-mapped Reference Creation.
  11. Checklist-based Performance Assessment using Clear
  12. Evaluation Process
  13. Generating Checklist-Mapped Model Responses.
  14. Assessing Response Quality using Checklists.
  15. Evaluation Component Validation
  16. ...and 129 more sections

Figures (11)

  • Figure 1: Pipeline of Clear. The example shown is from task T1: multi-document legal case summarization. The checklist mapper takes as input the model output (or human-written reference) and extracts checklist items according to the rubric. Checklists of the model output and the reference are compared at the item level, and the results are subsequently aggregated into the final scores.
  • Figure 2: F1 score vs. coverage of checklist items (i.e., the percentage of checklist items that are covered in the generation regardless their correctness).
  • Figure 3: Correlation of different model combinations with GPT-4o judgments averaged over all the tasks.
  • Figure 4: Regression analysis between the average performance along each task and the average correlation of judgments with GPT-4o assignments
  • Figure 5: Model performance across various levels of reasoning complexity.
  • ...and 6 more figures