Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments

Naghmeh Farzi, Laura Dietz

TL;DR

This work introduces an exam-based evaluation paradigm that replaces traditional relevance judgments with the EXAM Answerability Metric, defined by whether a system’s response enables answering a per-query exam bank. It formalizes a three-phase workflow—GenQ question bank generation, automated grading with LLM-based answer checking or self-rating, and two metrics (EXAM Cover and EXAM Qrels) compatible with trec_eval—while enabling human-in-the-loop question design. Empirical results across CAR Y3 and TREC DL show strong leaderboard correlations, often matching or exceeding relevance-prompt baselines, and demonstrate the approach’s robustness to generated question banks. The framework promises reusable test collections, post-hoc expansion, and practical cost reductions for evaluating future retrieval and generation systems, with clear paths for further human-in-the-loop refinements and broader adoption in IR evaluation tracks.

Abstract

Current IR evaluation is based on relevance judgments, created either manually or automatically, with decisions outsourced to Large Language Models (LLMs). We offer an alternative paradigm, that never relies on relevance judgments in any form. Instead, a text is defined as relevant if it contains information that enables the answering of key questions. We use this idea to design the EXAM Answerability Metric to evaluate information retrieval/generation systems for their ability to provide topically relevant information. We envision the role of a human judge to edit and define an exam question bank that will test for the presence of relevant information in text. We support this step by generating an initial set of exam questions. In the next phase, an LLM-based question answering system will automatically grade system responses by tracking which exam questions are answerable with which system responses. We propose two evaluation measures, the recall-oriented EXAM Cover metric, and the precision-oriented EXAM Qrels metric, the latter which can be implemented with trec_eval. This paradigm not only allows for the expansion of the exam question set post-hoc but also facilitates the ongoing evaluation of future information systems, whether they focus on retrieval, generation, or both.

An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments

TL;DR

This work introduces an exam-based evaluation paradigm that replaces traditional relevance judgments with the EXAM Answerability Metric, defined by whether a system’s response enables answering a per-query exam bank. It formalizes a three-phase workflow—GenQ question bank generation, automated grading with LLM-based answer checking or self-rating, and two metrics (EXAM Cover and EXAM Qrels) compatible with trec_eval—while enabling human-in-the-loop question design. Empirical results across CAR Y3 and TREC DL show strong leaderboard correlations, often matching or exceeding relevance-prompt baselines, and demonstrate the approach’s robustness to generated question banks. The framework promises reusable test collections, post-hoc expansion, and practical cost reductions for evaluating future retrieval and generation systems, with clear paths for further human-in-the-loop refinements and broader adoption in IR evaluation tracks.

Abstract

Current IR evaluation is based on relevance judgments, created either manually or automatically, with decisions outsourced to Large Language Models (LLMs). We offer an alternative paradigm, that never relies on relevance judgments in any form. Instead, a text is defined as relevant if it contains information that enables the answering of key questions. We use this idea to design the EXAM Answerability Metric to evaluate information retrieval/generation systems for their ability to provide topically relevant information. We envision the role of a human judge to edit and define an exam question bank that will test for the presence of relevant information in text. We support this step by generating an initial set of exam questions. In the next phase, an LLM-based question answering system will automatically grade system responses by tracking which exam questions are answerable with which system responses. We propose two evaluation measures, the recall-oriented EXAM Cover metric, and the precision-oriented EXAM Qrels metric, the latter which can be implemented with trec_eval. This paradigm not only allows for the expansion of the exam question set post-hoc but also facilitates the ongoing evaluation of future information systems, whether they focus on retrieval, generation, or both.
Paper Structure (31 sections, 3 equations, 2 figures, 9 tables)

This paper contains 31 sections, 3 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Task Statement.
  3. Contributions.
  4. Related Work
  5. LLM-based Relevance Assessment
  6. Nugget-based Evaluation
  7. Evaluation with Test Questions
  8. LLMs, Passages, and Questions
  9. Difference to Original EXAM sander2021exam
  10. Approach
  11. EXAM Question Banks
  12. Generating EXAM Question Banks (GenQ)
  13. Manual Verification and Refinement of EXAM Question Banks
  14. Manual Question Banks
  15. EXAM Grading
  16. ...and 16 more sections

Figures (2)

  • Figure 1: EXAM approach. Left: The only information available to the system that is being evaluated are queries and, optionally, a text corpus. The system response can be a ranking of passages, or a generated response that will be segmented into passages (blue). Passages from all systems will be pooled for assessment, and additional passages can be added at a later if needed. Right: The EXAM evaluation system uses three phases detailed in Section \ref{['sec:approach']}. For each query, an exam question bank is developed, which can be modified later in an iterative fashion (purple). All passages from the system response (e.g., p1, p2, p3) are graded based on which questions (q1?, q2?, ..., q5?) can be correctly answered with the passage text (red). We support two modes: one where answers are verified against an answer key (depicted as check marks), or by having an LLM self-rate the answerability on a scale from 0 to 5. The EXAM evaluation scores are derived from these grades (green). The EXAM Cover score is based on how many questions are covered, as binary verification or via a minimum self-rating level. For EXAM Qrels a relevance file for trec_eval is derived, which is based on the coverage or best self-rating obtained by this passage in isolation. We provide a worked example in Section \ref{['sec:worked-example']}
  • Figure 2: Comparing TREC CAR Y3 leaderboards, our EXAM metrics show strong correlation with the official TREC leaderboard. Methods are ordered by their official leaderboard rank (filled circles). Methods without official ranks are placed according to their TQA EXAM Cover scores. Standard error bars given. For context, Sander's EXAM leaderboard (depicted as crosses) does not show as strong a correlation.