Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

Jena D. Hwang, Varsha Kishore, Amanpreet Singh, Dany Haddad, Aakanksha Naik, Malachi Hamada, Jonathan Bragg, Mike D'Arcy, Daniel S. Weld, Lucy Lu Wang, Doug Downey, Sergey Feldman

TL;DR

It is shown that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge.

Abstract

Recent advances have made long-form report-generating systems widely available. This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along with meta-evaluation frameworks that seek to validate these methods. Many of the meta-evaluations estimate an evaluation quality's by comparing its assessments against human pairwise preferences. Prior work, however, suggests that human pairwise preference may be overly simplistic and can fail to capture nuances of expert expectations. We conduct a case study in meta-evaluation for long-form QA benchmarks using ScholarQA-CS2, a benchmark designed for assessing retrieval-augmented deep-research QA in the scientific domain. We comprehensively validate the benchmark through human pairwise preference judgments, then critically examine the strengths, weaknesses, and confounders of this approach. We show that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge. Based on our findings, we offer practical guidelines for designing future meta-evaluations that better align evaluation methods, annotator expertise, and reporting practices. By surfacing these methodological challenges, we aim to advance evaluation standards for deep-research systems.

Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

TL;DR

It is shown that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge.

Abstract

Recent advances have made long-form report-generating systems widely available. This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along with meta-evaluation frameworks that seek to validate these methods. Many of the meta-evaluations estimate an evaluation quality's by comparing its assessments against human pairwise preferences. Prior work, however, suggests that human pairwise preference may be overly simplistic and can fail to capture nuances of expert expectations. We conduct a case study in meta-evaluation for long-form QA benchmarks using ScholarQA-CS2, a benchmark designed for assessing retrieval-augmented deep-research QA in the scientific domain. We comprehensively validate the benchmark through human pairwise preference judgments, then critically examine the strengths, weaknesses, and confounders of this approach. We show that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge. Based on our findings, we offer practical guidelines for designing future meta-evaluations that better align evaluation methods, annotator expertise, and reporting practices. By surfacing these methodological challenges, we aim to advance evaluation standards for deep-research systems.
Paper Structure (31 sections, 10 figures, 9 tables)

This paper contains 31 sections, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Current Status of Long-form Scientific Report Evaluation
  3. Evaluation Benchmarks.
  4. Meta-Evaluation.
  5. ScholarQA-CS2 Benchmark
  6. Meta-Evaluation via Comparisons to Human Pairwise Judgments
  7. Annotation Setup
  8. Annotators.
  9. Experimental Settings
  10. Setting 1: Overall Preference; Random assignment.
  11. Principal Findings
  12. Additional Findings
  13. Recommendations
  14. Conclusion
  15. Appendix: Experimental Details
  16. ...and 16 more sections

Figures (10)

  • Figure 1: Our experimental settings for meta-evaluation of ScholarQA-CS2. We investigate meta-evaluation that assesses agreement between evaluation score and expert preference ranking (Setting 1). We also investigate settings (Settings 2 & 3) that compare evaluation scores against metric-wise judgment annotation and control for annotator expertise level.
  • Figure 2: When evaluating answer relevance disagreements, near- and deep-experts choose to keep their own judgments at similar rates; however, when they do not, near-experts tend to defer to the LLM's judgment more often than deep-experts.
  • Figure 3: Expert preference ranking vs. model scores. Model score alignments with expert preferences vary substantially across annotators.
  • Figure 4: Expert preference ranking vs. expert's metric-wise scores obtained in annotations settings 2 & 3. The expert preferences alignments with their own scores vary substantially across annotators.
  • Figure 5: Human annotation is conducted as a 3-way comparison. The results are converted into pairwise judgments, and is compared to the model scores for score (dis)agreements.
  • ...and 5 more figures